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DEVELOPMENT  OF  CANDIDATE  CREW  COORDINATION  EVALUATION  METHODS  AND 
MATERIALS 


Introduction 

The  development  of  crew  coordination  training  and  evaluation 
methods  is  driven  primarily  by  the  need  to  reduce  Army  aviation 
accidents  and  increase  mission  effectiveness.  Aviation  accidents 
represent  a  major  cost  to  the  Army,  and  a  significant  portion  of 
these  accidents  can  be  attributed  to  crew  error.  The  U.S.  Ariry 
Safety  Center  and  US  Army  Research  Institute  (USARI)  review  of  FY 
1984  through  FY  1989  aircraft  accidents  revealed  that  crew  error 
directly  contributed  to  147  fatalities  and  $292  million  in 
aviation  accident  costs  across  the  Army.  A  significant  number  of 
these  accidents  were  caused  by  the  failure  of  experienced  and 
qualified  crew  members  to  effectively  coordinate  their  decisions 
and  actions  in  the  cockpit.  While  aviation  accidents  represent  a 
highly  visible  piece  of  the  problem,  degraded  mission 
effectiveness  can  also  be  attributed  to  a  similar  pattern  of  crew 
errors.  Research  conducted  by  the  US  Army  Research  Institute 
(USARI)  in  1990  with  UH-60  aircrews  suggested  that  significant 
improvements  in  mission  effectiveness  would  result  from  better 
crew  coordination.  Thus,  the  focus  of  this  training  development 
research  was  to  identify  training  and  evaluation  methods  that 
would  serve  to  reduce  crew  error  and  its  consequences. 

Crew  coordination  training  programs  have  existed  for  years 
within  the  commercial  aviation  industry.  Such  programs  have 
responded  to  similar  needs  within  commercial  aviation  to  reduce 
accident  rates.  While  much  of  the  training  was  similar  to  that 
needed  by  the  Army,  the  evaluation  of  crew  performance  lacked 
meaningful  standards.  As  a  result,  training  effectiveness  could 
be  measured  only  in  terns  of  attitudinal  changes,  rather  than 
behavioral  or  performance  changes .  This  lack  of  meaningful 
evaluation  standards  was  identified  by  the  Army  as  a  major 
weakness  of  previous  training  programs . 

At  the  same  time,  it  was  recognized  that  the  demands  of 
tactical  aviation  were  not  exactly  matched  with  those  of 
commercial  aviation.  For  example,  responding  to  an  emergency  at 
an  altitude  of  50  feet  under  night  vision  goggle  conditions 
requires  a  different  approach  from  that  taken  in  responding  to  an 
emergency  at  35,000  feet.  The  failure  of  previous  training 
programs  to  discuss  crew  coordination  issues  within  the  specific 
context  of  Army  aviation  tasks,  conditions,  and  standards 
represented  a  second  weakness  of  these  existing  programs. 

Hence,  it  was  concluded  that  a  new  approach  to  crew 
coordination  training  should  be  developed.  This  approach  would 
build  upon  the  successful  elements  of  previous  training  programs 
available  from  commercial  aviation  and  the  other  military 
services.  At  the  same  time,  the  new  training  program  had  to  both 
(1)  incorporate  meaningful  evaluation  standards  and  (2)  relate 
crew  coordination  ideas  to  the  specific  tasks,  conditions,  and 
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standards  associated  with  Army  flight  operations.  Finally,  the 
focus  of  training  and  evaluation  had  to  extend  beyond  the 
classroom  in  order  to  be  relevant  to  Army  aviators.  The  bottom 
line  measure  of  success  for  this  training  program  was  a  reduction 
in  crew  error-induced  accidents  and  degraded  mission 
effectiveness.  Hence,  training  and  evaluation  has  to  take  place 
in  the  cockpit  where  meaningful  measures  of  behavior  and 
performance  can  be  obtained.  It  was  this  combined  set  of  goals 
to  which  the  present  research  and  development  effort  responded. 

Over  the  last  .several  years,  USARI's  Aviation  Research  and 
Development  Activity  (ARDA)  has  been  conducting  a  program  of 
training  research  that  responds  to  the  Army's  need  for  better 
crew  coordination  training.  This  program  of  research  has  been 
conducted  in  close  cooperation  with  the  US  Arny  Aviation  Center 
(USAAVNC)  and  its  effort  to  revise  its  training  standards  to 
reflect  increased  emphasis  on  crew-level  performance.  The 
following  report  describes  the  most  recent  phase  of  this 
research,  orchestrated  by  USARI  ARDA  and  USAAVNC  experts  and 
technically  supported  by  Dynamics  Research  Corporation. 

Purpose 

This  report  documents  work  performed  by  Dynamics  Research 
Corporation  (DRC)  to  develop  candidate  methods  and  materials  for 
evaluating  Army  aircrew  coordination  skills  in  the  rotary  wing 
cockpit.  It  summarizes  for  ARIARDA  and  USAAVNC  the  past  research 
and  current  guidelines  used  to  develop  an  evaluation  package  for 
a  crew  coordination  training  validation  testbed.  The  report 
describes  how  the  evaluation  methods  and  materials  were  developed 
for  the  exportable  training  package  and  the  exportable  evaluation 
package.  By  design,  the  evaluation  methods  documented  in  this 
report  are  general  enough  for  use  in  evaluating  aircrew 
coordination  in  all  mission  and  type  Army _ aircraft _ or  simulators. 
The  report  provides  ARIARDA  and  USAAVNC  with  materials  and 
suggested  applications  and  improvements  that  can  be  used  to 
evaluate  initial,  refresher,  and  continuation  aircrew 
coordination  training  conducted  in  resident  courses  or  by  units 
in  the  field. 

Overview 

The  crew  coordination  training  validation  testbed  was 
central  to  developing  the  candidate  evaluation  methods  and 
materials  (see  Figure  1) .  Research  findings  from  a  1990  testbed, 
jointly  sponsored  by  the  USAAVNC  and  the  USARI  were  advanced 
within  the  USAAVNC 's  current  guidelines  on  crew  coordination  to 
develop  the  evaluation  package  used  in  the  testbed.  Research 
materials  used  in  the  testbed  evaluation  package  were  reviewed 
based  on  lessons  learned  and  USAAVNC  guidelines  that  evaluation 
procedures  must  be  readily  useable  by  units  in  the  field. 
Candidate  evaluation  materials  from  the  Validation  Testbed 
Evaluation  Package  were  revised  and  incorporated  into  the  Field 
Exportable  Training  Package  which  is  designed  to  be  used  in 
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Figure  1.  Research  overview 


initial  and/or  refresher  training  and  the  Field  Exportable 
Evaluation  Package  which  is  to  be  used  for  continuation  training 
implemented  by  USAAVNC. 

Table  1  diagrams  the  development  and  relationship  between 
the  evaluation  materials  used  in  the  1990  and  1992  research 
testbeds  and  the  field  exportable  evaluation  materials  described 
in  this  report.  The  table  also  indicates  where  the  reader  can 
find  detailed  information  on  each  of  the  testbed  and  exportable _ 
evaluation  materials.  Information  on  the  levels  of  evaluation  is 
introduced  in  the  next  section. 
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Suiranary  of  Past  Research 

For  several  years  ARI  has  engaged  in  research  and 
development  of  a  new  training  and  evaluation  program  for  Army 
crew  coordination.  This  section  provides  a  brief  summary  of 
ARIARDA's  crew  coordination  research  program  and  major  findings 
from  past  research  conducted  by  ARIARDA  and  DRC. 

During  the  late  1970 's,  the  commercial  airlines  began  to 
apply  voice  and  flight  recorders  and  other  technologies  to 
examine  the  affect  of  human  performance  on  aircraft  accidents. 
Parallel  research  by  industry  and  government  agencies  extending 
well  into  the  1980 's  suggested  approaches  and  offered  programs 
for  teaching  cockpit  management  tasks.  The  military  services 
noted  the  airline  industry's  lessons  learned  and  began  using  the 
commercially-based  cockpit  resource  management  training  programs. 
The  Army  adapted  an  Aircrew  Training  Program  entitled  Dynamics  of 
Aircrew  Communication  and  Coordination  for  its  Initial  Entry 
Rotary  Wing  course  of  instruction  and  purchased  an  exportable 
course  for  units  in  the  field. 

While  the  commercial  airline  concepts  (e.g.,  attention, 
decision  making,  stress,  attitude,  and  risk  management)  were 
applicable  to  military  flight  operations  in  general,  their 
application  to  Army  aviation's  demanding  flight  environment  was 
not  obvious.  For  example,  an  airline  crew's  management  of  an 
emergency  at  35,000  feet  is  very  different  from  an  Army  aircrew's 
response  to  an  emergency  situation  50  feet  above  the  terrain 
operating  with  night  vision  goggles.  In  1987  the  Army's  Vice 
Chief  of  Staff  asked  ARI  to  initiate  research  aimed  at  reducing 
Army  accident  rates  through  better  personnel  selection,  training, 
and  system  design. 

Subsequent  analysis  of  Army  aviation  accidents, conducted 
jointly  by  USARI  and  the  U.S.  Army  Safety  Center  (USASC) ,  showed 
that  crew  coordination  errors  represented  a  major  category  of 
human  error  induced  accidents  in  aviation  operations  and  that  the 
Army  lacked  a  reliable,  objective  means  of  assessing  crew 
coordination  in  terms  of  safety  and  crew  performance.  Given 
these  findings  and  recognition  of  the  disparity  between 
commercial  and  Army  aviation  flight  environments,  ARIARDA 
initiated  a  long-range  research  program  in  two  phases  (FY  89-93 
and  FY  94-99)  to: 

1.  diagnose  specific  human  error  factors  prevalent  in  Army 
aviation  and  ground  operation  accidents,  and 

2.  identify  promising  strategies  for  reducing  such  accident 
causation  factors  through  improved  soldier  selection,  training, 
leadership,  and  organization. 

In  June  1989,  ARI  tasked  DRC  to  develop  measures  for 
evaluating  crew  coordination  training  effectiveness  and  then  in 
February  1990  ARI  tasked  DRC  to  translate  the  candidate  measures 
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into  techniques  and  procedures  for  identifying  and  documenting 
crew  coordination  errors  in  accident  investigations.  Table  2  and 
the  remainder  of  this  section  summarize  the  major  findings  and 
products  resulting  from  previous  research  developed  by  ARI  and 
technically  supported  by  DRC  on  crew  coordination. 

In  1989,  ARI  developed  a  program  of  research  to  identify 
solutions  for  improving  and  evaluating  the  coordination  and 
performance  of  rotary  wing  aircrews.  Technically  supported  by 
DRC,  a  portion  of  this  research  involved  experimental  evaluation 
of  aircrew  performance  in  the  UH-60  Flight  Simulator  to  determine 
specific  relationships  between  crew  coordination  behaviors  and 
safe  and  effective  mission  performance. 

Major  findings  of  this  research  showed  the  worth  of  flight 
simulators  to  crew  coordination  training  and  evaluation.  Flight 
simulators  allow  crews  to  execute  realistic  mission  scenarios 
under  a  variety  of  conditions  and  degrees  of  difficulty, 
including  high  risk  tasks,  and  provide  for  the  hands-on  component 
of  crew  coordination  training.  Similarly,  video  recording  of  the 
simulator  missions  supports  both  evaluation  and  training  by 
allowing  playback  and  review  of  crew  performance. 

Table  2 

Summary  of  ARIARDA  and  DRC  Research  (1989-1990) 


Research  Effort  Major  Findings  and  Products 


Development  of  Measures  of 
Crew  Coordination, 

August  1990 


Framework  for  training  and  evaluating  Army  Crew 
Coordination 

Flight  simulator  applications  to  training  and 
evaluation 

Video  recording  support  to  training  and 
evaluat ion 

Levels  of  evaluation  (attitude,  behavior, 
performance) 

Evaluation  measures  (attitude  questionnaire, 
behavioral  dimensions  and  rating  scale,  task 
performance) 

Evaluation  training  course 

Outline  of  training  course  for  aviators 


Development  of  Rating 
Instruments  and  Procedures 
for  Aviation  Mishap 
Investigation ,  September 
1990 


ATM  crew  coordination  elements- task  errors 
Crew  coordination  objectives-system 
inadequacies 

Proposed  revisions  to  DA  Pam  385-95 
Training  program  for  accident  investigators 


Results  of  the  Data 
Analysis  Army  Aircrew 
Coordination  Measures 
Testbed,  Conducted  Spring 
1990,  April  1991 


Relationship  between  measures  of  crew 
coordination  and  crew  performance 
Methodology  to  collect  mission  performance 
measures 
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Levels  of  Evaluation 


The  research  method  developed  to  assess  crew  coordination 
during  the  simulator  experiments  consisted  of  three  levels  of 
evaluation.  The  first  level  was  crewmember  attitude.  The 
linkage  between  attitude  and  cockpit  performance  had  been 
postulated  by  Helmreich  (Helmreich,  1984  and' Helmreich  et  al . , 
1986)  who  developed  the  Cockpit  Management  Attitudes 
Questionnaire  (CMAQ)  for  the  National  Aeronautics  and  Space 
Administration  (NASA) . 

The  second  level  of  evaluation  was  based  on  observable  crew 
behavior.  Helmreich  and  Wilhelm  (1987)  had  developed  and  tested 
an  instrument  called  the  Line/LOFT  (Line  Oriented  Flight 
Training)  Worksheet,  which  used  17  dimensions  of  behavior  to 
solicit  expert  ratings  of  crew  performance.  Applications  of  the 
Line/LOFT  Worksheet  had  shown  it  to  be  useful  in  crew 
coordination  proficiency  evaluations. 

The  third  level  of  evaluation  was  performance.  The  Army's 
Aircrew  Training  Manual  (ATM)  tasks,  evaluated  in  the  Annual 
Proficiency  and  Readiness  Test  (APART)  program,  were  individual 
rather  than  crew  tasks.  Revisions  to  include  crew  coordination 
in  selected  ATM  tasks  were  made.  Crew  performance  was  also 
examined  by  reviewing  video  tapes  of  completed  mission  outcomes . 

Evaluation  Measures 

The  crew  coordination  measurement  suite  paralleled  the  three 
levels  of  evaluation.  The  first  component.  Army  Aviation 
Crewmember  Questionnaire,  was  based  on  the  military  version  of 
the  Cockpit  Management  Attitudes  Questionnaire  (CMAQ)  (Helmreich, 
1984,  Helmreich  and  Wilhelm,  1987)  .  Several  aspects  of  the  basic 
CMAQ  were  altered  to  align  with  the  Ariry's  doctrine,  terminology, 
and  proposed  crew  coordination  objectives  to  produce  a  45  item 
attitude  measurement  instrument.  The  attitude  instrument  was 
administered  to  crewmembers  and  instructor  pilot  evaluators. 

The  Aircrew  Coordination  Evaluation  (ACE)  Checklist  was  the 

second  component  of  the  measurement  suite.  The  NASA/University 
of  Texas  military  version  of  the  Line/LOFT  Worksheet  (Helmreich 
and  Wilhelm,  1987)  was  the  basis  for  the  ACE  Checklist.  The 
Line/LOFT  was  modified  to  align  its  behavioral  dimensions  with 
the  Army's  crew  coordination  behavioral  objectives.  Written 
descriptions  were  prepared  for  each  crew  coordination  dimension 
and  its  associated  rating  scale  anchors  for  poor,  acceptable,  and 
superior  performance.  The  resulting  ACE  Checklist  of  19  crew 
coordination  dimensions  was  administered  to  UH-60  crews  by 
specially  trained  military  instructor  pilots. 

Revised  Aircrew  Training  Manual  (ATM)  tasks  was  the  third 
evaluation  measure.  Aircrew  Training  Manual,  Utility  Helicopter 
UH-60  (Training  Circular  1-212,  1988)  was  reviewed  to  identify 
tasks  with: 
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1.  significant  aircrew  coordination  behaviors  underlying 
their  successful  accomplishment,  or 

2.  frequent  citations  in  aviation  accidents  which  could 
have  been  influenced  by  inclusion  of  aircrew  coordination 
behaviors . 

Selected  tasks  were  rewritten  to  include  crew  coordination 
behaviors  in  the  task's  description,  conditions,  and  standards. 
Detailed  standards  included  in  ATM  task  1071,  Perform  as  a  Crew 
Member,  were  referenced  by  other  tasks  with  crew  coordination 
behaviors.  The  ATM  grade  slip  was  modified  to  allow  separate 
evaluation  of  the  flight  skill  and  aircrew  coordination 
components  of  the  revised  ATM  tasks. 

The  Development  of  Measures  of  Crew  Coordination  research 
effort  included  developing  and  trying  out  a  course  of  instruction 
on  crew  coordination  principles  and  evaluation  techniques.  The 
course  was  designed  to  provide  instructor  pilot  evaluators  with 
the  information  necessary  to  understand  and  implement  the 
evaluation  procedures. 

During  a  U.S.  Army  Safety  Center  (USASC)  and  ARIARUA  leview 
of  FY  1984  through  FY  1989  aircraft  accidents,  it  became  apparent 
that  the  basic  framework  used  to  develop  measures  of  crew 
coordination  could  potentially  be  used  to  develop  an  improved 
methodology  for  investigating  crew  coordination  errors  during 
aviation  accident  investigations.  In  February  1990,  ARIARDA 
tasked  DRC  to  translate  candidate  measures  into  techniques  and 
procedures  for  potential  use  by  USASC  accident  investigation 
teams  to  identify  and  document  crew  coordination  errors  and 
system  inadequacies. 

Aviation  accident  case  histories  were  subjected  to  an  in 
depth  analysis  to  determine  to  what  degree,  if  any,  aircrew 
coordination  errors  were  a  factor.  The  analysis  resulted  in 
definitions  for  nine  task  error  categories  and  four  system 
inadequacies  or  underlying  causes  that  permit  a  crew  coordination 
task  error  to  become  an  accident  cause.  These  definitions  and 
their  relationships  were  incorporated  into  a  proposed  Aircrew 
Coordination  Accident  Investigator  Handbook  designed  to  be  used 
by  the  Human  Factors  member  of  USASC  accident  investigation 
boards.  The  Handbook  provided  guidance,  format,  and  definitions 
required  to  identify  and  document  aircrew  coordination  task 
errors  resulting  in  or  relevant  to  aircraft  accidents. 

DRC  personnel  validated  the  Handbook  and  gained  experience 
in  applying  the  new  crew  coordination  definitions  by  actually 
participating  in  USASC  aircraft  accident  investigations. 

Insights  from  the  accident  investigation  process  and  interviews 
with  accident  crewmembers  indicated  that  the  crew  coordination 
definitions  adequately  addressed  all  crew  actions,  especially  the 
interactions  among  crewmembers.  This  direct  feedback  from  the 
field  helped  more  precisely  define  the  aircrew  coordination 
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principles  and  provide  specific  examples  of  observable  crew 
performance.  The  definitions  provided  a  practical  means  to  link 
primary  and  secondary  aircrew  coordination  errors  while 
separating  individual  errors  from  crew  errors  contributing  to  an 
accident.  This  use  of  the  definitions  aided  the  development  of 
clearly  written  descriptions  of  crew  behavior  levels  for  rating 
crew  performance.  The  importance  of  crew  climate,  workload 
distribution,  situation  awareness,  and  the  role  of  nonrated 
crewmembers  was  punctuated  by  the  accident  investigation 
experience. 

DRC's  research  aligned  the  accident-based  task  error  and 
system  inadequacy  definitions  with  the  research-based  ACE 
dimensions  and  crew  coordination  objectives.  Once  these 
relationships  between  accidents,  task  errors,  and  system 
inadequacies  were  established,  USAAVNC  used  these  notions  in  the 
revised  Training  Circulars  as  crew  coordination  elements  and  crew 
coordination  objectives. 

The  final  report  on  the  1990  testbed  provided  the  results  of 
a  data  analysis  that  examined  the  functional  relationships 
between  aircrew  coordination  attitudes,  behaviors,  and 
performance.  The  analysis  concluded  that  crew  coordination 
attitudes,  behavior  and  performance  were  all  related  to  each 
other  and  to  mission  related  performance  variables.  The  report 
included  a  comparison  of  crew  performance  based  on  variables 
identified  by  reviewing  video  tapes  of  completed  missions.  These 
mission  performance  variables  illustrated  the  feasibility  of 
developing  mission-related  crew  performance  measures. 

Findings  of  this  research  combined  with  research  conducted 
in-house  by  the  U.S.  Army  led  to  the  identification  of  several 
areas  in  which  aircrew  training  and  evaluation  needed  to  be 
improved : 

1.  cockpit  communication  procedures, 

2.  management  of  crew  workload  in  the  cockpit, 

3.  cross-monitoring  of  crewmembers, 

4.  team  relationships  and  crew  climate,  and 

5.  mission  planning  and  rehearsal. 

These  five  areas  are  published  in  the  Army's  Aircrew 
Training  Program  (Aircrew  Training  Program  Commander's  Guide  to 
Individual  and  Crew  Training,  Training  Circular  1-210)  and 
Aircrew  Training  Manuals  as  objectives  for  crew  coordination. 
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Summary  of  Guidelines  for  Crew  Coordination 

In  1991,  theUSAAVNC  established  a  special  crew  coordination 
Working  Group  to  incorporate  crew  coordination  considerations 
into  the  Army's  Aircrew  Training  Program.  The  Working  Group 
guided  the  inclusion  of  major  research  findings  and  products  such 
as  the  crew  coordination  objectives  and  crew  coordination 
elements  into  program-wide  Training  Circulars  and  aircraft 
specific  Aircrew  Training  Manuals  (ATM) .  They  also  revised  all 
ATM  tasks  to  include  crew  coordination  considerations.  The  new 
Training  Circulars  provided  broad  guidance  on  what  to  do  but  did 
not  address  the  implementing  details  such  as  how  to  train  unit 
aircrews.  It  was  recognized  that  the  areas  of  how  to  train  crews 
on  the  newly  announced  crew  coordination  principles  and  how  to 
evaluate  crew  performance  were  too  general  to  ensure  that  the 
crew  coordination  initiative  would  succeed  nor  would  it  be  able 
to  achieve  any  degree  of  standardization.  USAAVNC  subsequently 
requested  USARI  to  develop  and  validate  an  exportable  training 
package  for  instructing  experienced  aviators  in  the  field  on  the 
new  crew  coordination  principles  and  ATM  standards. 
Simultaneously,  USAAVNC  began  work  to  incorporate  this  same 
material  into  the  various  schoolhouse  training  courses  for  new 
aviators . 

This  section  provides  a  summary  of  the  USAAVNC 's  written  and 
verbal  guidelines  that  transitioned  the  research  findings  and 
products  from  ARIARDA/DRC ' s  1989-1991  program  into  the  validation 
testbed  evaluation  package.  The  prima2q/  sources  of  guidelines 
were  the  new  Training  Circulars  (i.e..  Commander's  Guide  and 
aircraft  specific  Aircrew  Training  Manuals)  and  the  Crew 
Coordination  Working  Group.  The  Working  Group,  composed  of 
representatives  from  the  USAAVNC 's  Directorate  of  Training  and 
Doctrine,  Directorate  of  Evaluation  and  Standardization,  Aviation 
Training  Brigade,  Center  Safety,  and  ARIARDA,  met  with  DRC ' s  crew 
coordination  project  staff  regularly  to: 

1.  provide  Training  Circular  updates, 

2.  clarify  written  guidelines, 

3.  review  project  progress, 

4.  participate  in  resolving  issues,  and 

5.  issue  detailed  guidelines. 

Aircrew  Training  Program  Commander's  Guide 

The  newly  published  Aircrew  Training  Program  Commander's 
Guide  to  Individual  and  Crew  Training,  Training  Circular  1-210, 
describes  Army  Aviation's  crew  coordination  training  and 
evaluation  philosophy  and  defines  crew  coordination: 
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Army  Aviation's  position  on  crew  coordination  is  that 
"Planning,  preflight,  and  in-flight  tasks  involve  the 
cooperative  effort  of  all  crewmembers.  The  prescribed 
tasks,  conditions,  standards,  and  descriptions  explain  each 
crewmember's  responsibility  for  the  successful  completion  of 
maneuvers.  Each  crewmember  must  understand  the  actions  and 
directives  of  the  other  crewmembers.  This  enhances  crew 
coordination  and  unit  interoperability  and  helps  to  prevent 
accidents  caused  by  crew  error"  (p.  iv) . 

The  Training  Circular  specifies  that  "Aircrew  training 
manuals  (ATMs)  published  after  this  manual  will  incorporate 
the  concept  of  crew  coordination  and  training  as  a  crew 
rather  than  training  exclusively  as  an  individual"  (p.  iv) . 

Crew  coordination  is  defined  as  "a  set  of  principles, 
attitudes,  procedures,  and  techniques  which  transforms 
individuals  into  an  effective  crew"  (p.  1-4) . 

The  Commander's  Guide  also  documents  the  five  program-wide 
objectives  for  crew  coordination  and  eight  crew  coordination 
elements  identified  in  past  research.  The  crew  coordination 
objectives  and  elements  are  repeated  in  each  ATM  as  guidance  to 
implement  crew  coordination  training  and  evaluation.  These 
guidelines  plus  other  information  in  the  Commander's  Guide  on 
selecting  crew  tasks  for  evaluation,  managing  crew  risk,  and 
tracking  crew  performance  provided  the  basis  for  developing  the 
validation  testbed  evaluation  package. 

Revised  Aircrew  Training  Manuals  (ATMs) 

During  1992  new  ATMs  incorporating  crew  coordination 
guidance  were  being  published  and  distributed.  Working  Group 
members  provided  the  project  staff  with  up-to-date  draft  and 
final  versions  of  each  publication.  This  arrangement  allowed  for 
early  consideration  of  the  more  detailed  guidelines  being  issued 
to  the  field  on  how  to  implement  crew  coordination. 

The  revised  ATM  tasks  explicitly  identified  crew 
coordination  standards  (e.g.,  correctly  perform  crew  coordination 
actions)  and  described  individual  crewmember  duties  (e.g.,  pilot 
flying,  pilot  not  flying,  crew  chief,  etc.).  Additionally,  the 
crew  coordination  elements  implied  in  each  ATM  task  indicated  the 
need  to  evaluate  multiple  aspects  of  crew  coordination  within  a 
single  ATM  task. 

Evaluation  principles  and  grading  considerations  in  each  ATM 
stated  that,  "The  guidelines  for  evaluating  crew  coordination  are 
not  based  on  objective  criteria;  for  example,  distance  or 
degrees.  Rather  they  are  based  on  a  subjective  analysis  of  how 
effectively  a  crew  performs  together  to  accomplish  a  series  of 
tasks.  The  subjective  analysis  is  as  important  as  the  objective 
evaluation  of  the  more  definitive  measurable  tasks.  The 
evaluator  measures  crew  coordination  skills  on  the  basis  of 
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subjective  judgment,  which  is  more  difficult  than  objectively 
measuring  the  skill  to  accomplish  a  specific  task"  (chapter  8, 
ATMs) .  Guidelines  on  conducting  the  subjective  analysis  of  crew 
coordination  skills  required  that  evaluators  consider  two 
criteria: 

1.  how  effectively  does  each  crewmember  communicate,  and 

2.  how  effectively  does  each  crewmember  sequence  and  time 
critical  actions  to  complete  a  task  successfully. 

Brief  explanations  of  these  still  individual-oriented 
criteria  included  some  language  from  the  ATM  crew  coordination 
elements  but  did  not  directly  refer  to  the  elements  or  provide 
clear  definitions  to  reduce  subjectivity. 

Grade  slips  in  the  new  ATMs  provided  for  recording 
performance  at  the  rated  and/or  nonrated  crewmember  level 
(Maneuver/ Procedure  Grade  Slip)  and  at  the  battle-rostered  crew 
level  (Battle-Rostered  Crew  Evaluation/Training  Grade  Slip) .  The 
Maneuver/ Procedure  Grade  Slip  covered  all  ATM  tasks  while  the 
Battle-Rostered  Crew  Evaluation/Training  Grade  Slip  allowed 
commanders  to  designate  mission  essential  crew  tasks.  The 
aircrew  grading  system  specified  the  use  of  satisfactory  (S), 
unsatisfactory  (U) ,  or  not  applicable  (NA)  for  both  ATM  tasks  and 
commander  designated  crew  tasks. 

Crew  Coordination  Working  Group 

The  Working  Group  provided  the  background  information  and 
detailed  guidelines  necessary  to  develop  and  document  a  method 
for  evaluating  crew  coordination  that  implemented  the  new 
Training  Circulars .  Many  of  the  Working  Group  members  had 
directly  participated  in  developing  the  new  Commander's  Guide  and 
revised  ATMs.  They  were  familiar  with  the  background  and 
intended  meaning  of  the  concepts  and  guidelines  contained  in  the 
new  publications.  The  Working  Group  was  fully  qualified  and 
authorized  to  both  clarify  and  issue  guidelines  to  develop  an 
acceptable  method  for  evaluating  crew  coordination  performance. 

Initial  discussions  with  the  Working  Group  clarified  major 
points  that  guided  development  of  the  evaluation  approach.  The 
evaluation  method  had  to  be: 

1.  capable  of  measuring  the  effect  of  crew  coordination 
training  on  both  flight  safety  and  effective  mission  performance, 

2.  consistent  with  the  new  Commander's  Guide  and  the  new 
ATMs  for  all  Army  aircraf t--rotarY  and  fixed  wing,  and 

3.  suitable  for  use  in  visual  simulators  and  in  the 
aircraft  itself  because  visual  simulators  were  not  available  for 
all  platforms. 
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A  priority  for  the  Working  Group  was  to  supplement  the 
Commander's  Guide  and  ATMs  with  the  seminal  information  needed  to 
effectively  teach  and  evaluate  crew  coordination.  The  Working 
Group  reviewed  contractor  and  other  service  developed  courses  of 
instruction  and  research  reports  on  crew  coordination.  Findings 
in  ARIARDA/DRC ' s  past  research  (Simon,  1991)  indicated  a  strong 
correlation  between  crew  coordination  behaviors  and  task 
performance.  The  Working  Group  made  extensive  use  of  the 
information  contained  in  the  Simon  (1990,1991)  research  report. 
Initial  actions  compared  the  aircrew  coordination  elements 
described  in  the  new  ATMs  with  the  crew  coordination  behaviors  or 
dimensions  from  the  Aircrew  Coordination  Evaluation  (ACE) 
Checklist  used  in  the  1990  experiment. 

Over  a  series  of  meetings,  the  Working  Group  guided  project 
staff  to  add,  delete,  and  combine  the  ACE  behavior  titles  and 
definitions  to  produce  a  set  of  13  crew  coordination  behaviors 
called  Basic  Qualities.  Written  descriptions  of  superior, 
acceptable,  and  very  poor  performance  were  prepared  for  each 
Basic  Quality  definition.  These  behavioral  descriptions  were 
assigned  values  of  seven,  four,  and  one,  respectively,  to  serve 
as  anchors  for  a  seven-point  rating  system.  The  Working  Group 
specified  that  the  Basic  Qualities  and  the  behaviorally  anchored 
rating  system  (BARS)  be  central  to  the  evaluation  method. 

From  the  outset,  the  Working  Group  stressed  that  the  Army 
train  crew  coordination  in  the  same  manner  as  it  planned  to 
evaluate  crew  performance.  The  Basic  Qualities  composing  the  new 
ACE  Checklist  were  then  aligned  and  sequenced  with  the  previously 
approved  crew  coordination  objectives  for  use  in  organizing  the 
training  course  content.  This  action  effectively  placed  crew 
coordination  training  and  evaluation  on  a  common  framework  of 
crew  coordination  objectives,  Basic  Qualities,  and  performance 
standards.  By  design,  training  and  evaluation  share  common 
terms,  definitions,  and  descriptions  of  crew  behaviors  that 
implement  the  Aritiy's  new  Aircrew  Training  Manuals. 

Project  staff  met  with  the  Working  Group  to  discuss  the 
adequacy  of  the  new  ATM  grading  procedures  and  grade  slips  to 
evaluate  crew  coordination  performance.  The  Working  Group 
specified  that  the  Maneuver/ Procedure  Grade  Slip  for  each 
aircraft  be  used  to  evaluate  ATM  tasks  but  agreed  to  consider 
modifications  if  needed  to  support  the  use  of  the  Basic  Qualities 
and  behaviorally  anchored  rating  system  (BARS)  measures.  Project 
staff  proposed  minor _ changes  to  the  grade  slip  and  grading 
procedures  to  explicitly  evaluate  the  crew  coordination  skills 
included  in  all  ATM  tasks .  The  Working  Group  approved  the 
modifications  but  restricted  the  use  of  the  Aircrew  Coordination 
Training  Grade  Slip  to  initial  and  refresher  crew  coordination 
training  only.  The  Battle-Rostered  Crew  Evaluation/Training 
Grade  Slip  would  be  used  to  evaluate  crew  coordination 
continuation  training.  The  Working  Group  specified  that 
instructor  pilots  and  unit  trainers  be  trained  on  the  evaluation 
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method  and  its  use  in  the  crew  coordination  course  of 
instruction.  Table  3  summarizes  the  guidelines  for  evaluating 
crew  coordination. 


Table  3 

Summary  of  Guidelines  for  Evaluating  Crew  Coordination 


Evaluation  Topic 


Guidance  (Source*) 


Philosophy 


Define  crew  coordination  (CG) 
Emphasize  crew  rather  than  individual 
aviator  performance  (CG) 


Approach  •  Define  the  evaluation  process  (WG) 

•  Ensure  consistency  to  implement  new 
Commander's  Guide  and  ATMs  (WG) 

•  Measure  crew  coordination  training  effect 
on  flight  safety  and  mission  effectiveness 
(WG) 

•  Ensure  suitability  for  use  in  all  Army 
aircraft  (rotary  and  fixed  wing)  and  visual 
simulators  (WG) 

•  Focus  on  relationship  between  behaviors 
and  performance  (WG) 

Attitude  Measure  •  Limit  use  to  validation  testbed  evaluation 

paclcage  (WG) 


Behavioral  Measure 


Relate  ACE  behavioral  dimensions  to  ATM 
crew  coordination  elements  and  objectives 
(WG,  ATM) 

Add,  delete,  and  combine  behavioral  titles 
and  definitions  (WG) 

Sequence  crew  coordination  behaviors  in 
mission  execution  order  (WG) 

Refer  to  crew  coordination  behaviors  as  Basic 
Qualities  (WG) 

Include  example  ATM  taslcs  in  Basic  Quality 
definitions  (WG,  ATM) 

Incorporate  the  7-point  behaviorally  anchored 
rating  scale  (WG) 

Revise  behavioral  anchor  written  descriptions 
(WG) 


Grade  Slips 


Modify  the  aircraft  maneuver  procedure  grade 
slip  and  identify  it  as  the  Aircrew 
Coordination  Training  Grade  Slip  (WG,  ATM) 
Limit  use  of  the  Aircrew  Coordination 
Training  Grade  Slip  to  crew  coordination 
initial  and/or  refresher  training  (WG) 

Include  Battle-Rostered  Crew  Evaluation/ 
Training  Grade  Slip  for  overall  performance 
and  evaluator  comments  (WG,  ATM) 

Incorporate  S+,  S,  S-,  and  U  expanded  grading 
system  (WG) 


14 


Table  3 


Svammary  of  Guidelines  for  Evaluating  Crew  Coordination  (Cont.) 


Evaluation  Topic 


Guidance  (Source*) 


Scenarios 

•  Develop  realistic  tactical  scenarios  for 
evaluation  missions  (WG) 

•  Provide  video  recording  and  playback  to 
assist  evaluators  (WG) 

•  Provide  guidelines  for  scenario  development 
(WG) 

Mission  Performance 

Measures 

•  Ensure  objective  comparison  across  crews  to 
support  overall  mission  performance  (WG) 

Evaluator  Workbook 

•  Ensure  that  exceptions  to  ATM  evaluation 
procedures  are  clearly  stated  (WG,  ATM) 

Interaction  with  Training 
Course 

•  Train  instructor  pilots  and  unit  trainers  on 
evaluation  method  and  materials  \WG) 

•  Limit  pre-  and  post-training  evaluations  to 
instructor  pilots  (WG) 

Exit  Interviews 

•  Provide  for  individual  and  group  formats  (WG) 

•  Limit  use  to  valiation  testbed  evaluation 
package  (WG) 

♦Commander's  Guide  (CG) ,  Aircrew  Training  Manual  (ATM),  Working  Group  (WG) 


Development  of  the  Validation  Testbed  Evaluation  Package 

This  section  describes  the  major  activities  undertaken  to 
develop  the  validation  testbed  evaluation  package  and  presents 
the  evaluation  products  (see  Figure  2). 

Evaluation  Requirements  and  Approach 

Initially,  the  crew  coordination  validation  testbed  was  to 
confirm  the  evaluation  method  and  materials  using  AH-64 
(priority)  and  UH-60  crews  and  visual  simulators.  The  crew 
coordination  Working  Group  subsequently  deleted  the  AH-64  from 
the  testbed  due  to  scheduled  modifications  to  the  combat  mission 
simulators  and  the  unavailability  of  crews  in  the  field  units. 
Although  the  products  were  developed  to  evaluate  utility, 
helicopter  crew  operations  in  the  validation  testbed,  they 
accommodate  all  Army  aircraf t--rotary  and  fixed  wing. 

Based  on  Working  Group  guidelines,  methods  and  measures 
introduced  in  ARIARDA/DRC ’ s  past  research  (Simon,  August  1990  and 
Simon,  1991)  were  selected  as  start  points.  DRC  designed  a 
methodology  that  measures  the  difference  in  performance  within  a 
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Figure  2.  Evaluation  methods  and  materials  development. 

group  of  crews  before  and  after  receiving  crew  coordination 
training.  A  testbed  using  Army  aviation  crews  in  their  normal 
training  environment  was  selected  to: 

1.  validate  the  evaluation  method  itself,  that  is,  unit 
instructor's  understanding  and  ability  to  apply  the  method  to 
produce  objective,  reliable  evaluations  of  crew  coordination,  and 

2.  validate  the  Army's  newly  designed  crew  coordination 
training  course. 

Three  levels  of  evaluation  form  the  basis  of  the  methodology 
(see  Figure  3) .  At  the  first  level,  the  methodology  includes  a 
questionnaire  to  assess  individual  crewmember  attitudinal  changes 
resulting  from  participation  in  crew  coordination  training.  The 
second  level  addresses  behavioral  changes  in  the  cockpit  and 
includes  independent  measures  consisting  of  behaviorally  anchored 
rating  scales  and  instructor  pilot  rating  forms.  The  rating 
scales  can  be  applied  to  all  Army  aircraft,  whereas  the  rating 
forms  are  based  on  the  new  ATMs  for  each  aircraft .  At  the  third 
level,  the  methodology  addresses  flight  safety  and  combat  mission 
effectiveness . 

Attitudinal  Measure 

An  attitudinal  measure  was  included  in  the  validation 
testbed  evaluation  process  for  two  reasons:  to  further  explore 
the  possible  relationship  between  attitudes  and  task  performance 
and  to  measure  the  pre-  and  post-training  effect  on  crewmember 
attitudes  toward  crew  coordination. 
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Figure  3.  Levels  of  evaluation. 

The  attitudinal  questionnaire,  Army  Cockpit  Management 
Attitudes  Questionnaire  (CMAQ) ,  used  in  the  1990  experiment 
served  as  the  basis.  The  45  item  CMAQ  was  critically  analyzed 
and  revised.  Instructions  were  revised  to  stress  the  importance 
of  the  Crew  Coordination  Program  on  aviation  safety  and  mission 
effectiveness.  The  entire  instrument  was  edited  to  accommodate 
all  Army  aircraft,  remove  references  to  commercial  aviation,  and 
incorporate  current  doctrine  and  training  literature  terms  like 
crew  readiness  level  and  after-action  review. 

Analysis  of  the  reliability  of  each  item  in  the  previous 
instrument  was  used  to  identify  and  eliminate  10  weak  questions. 
Factor  analysis  of  the  remaining  items,  including  questions  from 
the  NASA-University  of  Texas  attitude  survey  of  commercial 
airline  and  Air  Force  crews,  resulted  in  a  three-factor  model 
(Simon,  1991)  that  best  explained  the  variance  in  responses.  The 
number  of  questions  in  the  Communication  and  Coordination,  Shared 
Leadership,  and  Recognition  of  Stressor  Effects  factor  scles  was 
considered  when  revising  previous  items  and  creating  new  items. 
Previous  items  were  revised  and  new  items  were  created  to  address 
new  subject  areas  like  premission  planning  and  rehearsal  and 
after-action  review  that  were  being  included  in  the  Crew 
Coordination  Exportable  Training  Package  (Pawlik,  Simon,  Grubb,  & 
Zeller,  1992) . 

A  scoring  key  was  prepared  for  the  46  items  to  help  sequence 
and  balance  the  overall  instrument.  Additionally,  items  were 
balanced  within  factors  and  sequenced  to  avoid  concentrating  a 
series  of  questions  that  addressed  a  single  factor  in  one  area  of 
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the  questionnaire.  ARIARDA  reviewed  and  approved  the 
questionnaire.  The  final  attitudinal  measure  is  at  Appendix  A. 

Behavioral  Measures 


The  crew  coordination  Working  Group  endorsed  crew  behaviors 
as  the  principal  means  to  teach  and  evaluate  crew  coordination 
making  this  an  especially  important  measure.  Products  developed 
to  evaluate  crew  coordination  behaviors  include  behavioral 
measures,  a  rating  scale,  and  guidelines  on  how  to  rate  crew 
performance. 

The  initial  action  to  develop  behavioral  measures  was  to 
review  the  results  of  the  1990  experiment's  Aircrew  Coordination 
Evaluation  (ACE)  Checklist  (Simon,  1991) .  Reliabilities  were 
exceptionally  high  for  the  19  behavioral  items  found  in  the  1990 
ACE  Checklist.  Item  analysis  placed  each  of  the  crew 
coordination  behaviors  or  "dimensions"  into  one  of  five 
behavioral  domains,  that  is,  four  separate  domains  and  a  total. 
The  four  crew  coordination  domains  were: 

1.  establish/maintain  team  relationships, 

2.  cross  monitoring  of  crew  performance, 

3.  mission  information  exchange,  and 

4.  establish/maintain  reasonable  workload  levels. 

After  additional  research  and  analysis,  the  project  staff 
made  the  following  changes: 

1.  Add  mission  planning  and  rehearsal  as  a  domain. 

2.  Delete  ACE  dimensions  (3)  Inquiry/questioning 
practiced,  (15)  Overall  technical  proficiency,  (16)  Overall  crew 
effectiveness,  and  (17)  Overall  workload. 

3.  Combine  dimensions  (5)  Decisions  communicated  and 
acknowledged  and  (6)  Actions  communicated  and  acknowledged  into  a 
new  dimension  (5)  titled  Decisions  and  actions  communicated  and 
acknowledged. 

4.  Retitle  dimension  (18)  Conflict  resolution  to  read 
Resolution  of  disagreements. 

5.  Add  premission  rehearsal  accomplished  as  a  dimension. 

The  revised  behavioral  domains  and  set  of  ACE  behaviors  were 
compared  with  the  aircrew  coordination  objectives  and  elements 
described  in  the  new  ATMs.  Based  on  published  definitions, 
relationships  were  established  with  all  5  behavioral  domains  and 
11  of  the  15  ACE  dimensions. 
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The  Working  Group  guided  project  staff  to  add,  delete,  and 
combine  the  behavior  titles  and  definitions  to  produce  a  set  of 
13  crew  coordination  behaviors  called  Basic  Qualities.  The  Basic 
Qualities _ composing  the  new  ACE  Checklist  were  then  sequenced  and 
aligned  with  the  published  objectives  for  the  Army's  Crew 
Coordination  Program.  Figure  4  illustrates  the  relationship  of 
Basic  Qualities  to  ATM  crew  coordination  elements  and  crew 
coordination  program  objectives.  The  Working  Group  encouraged 
the  use  of  the  chart  in  the  training  course  to  enable  crewmembers 
to  recognize  and  understand  a  clear  linkage  or  "crosswalk"  among 
the  related  crew  coordination  terms . 

The  former  ACE  dimensions  were  revised  to  incorporate  the 
new  Basic  Quality  titles  and  structure.  Behavioral  measures  were 
created  for  the  new  Basic  Qualities,  that  is,  establish  and 
maintain  flight  team  leadership  and  crew  climate,  application  of 
appropriate  decision  making  techniques,  and  crew-level  after¬ 
action  review  accomplished.  Abbreviated  title  identifiers  were 
prepared  to  facilitate  use  of  the  Basic  Qualities  in  related 
evaluation  materials  like  grade  slips  and  evaluator  worksheets. 

The  Basic  Quality  definitions  and  descriptions  of  the  types 
of  behaviors  for  superior,  acceptable,  and  very  poor  levels  of 
performance  were  revised  and  extended  to  include  example  tasks 
from  the  new  ATMs.  The  written  descriptions  of  the  levels  of 
performance  for  each  Basic  Quality  definition  were  assigned 
values  of  seven,  four,  and  one  to  serve  as  anchors  for  a  seven- 
point  rating  system.  The  Working  Group  specified  that  the  Basic 
Qualities  and  the  behaviorally  anchored  rating  system  (BARS)  be 
central  to  the  evaluation  process.  The  numeric  rating  scale 
shown  below  was  developed  for  evaluators  to  assess  the  level  of 
behavior  that  crews  exhibited  for  each  Basic  Quality. 

Very  Accept-  Very 

Poor  Poor  Marginal  able  Good  Good  Superior 

1  2  3  4  5  6  7 

Guidelines  were  prepared  to  assist  evaluators  in  assessing 
how  well  a  crew  performed  each  Basic  Quality  in  relation  to  the 
ACE _ Checklist ‘ s  behavioral  anchors.  For  example,  in  completing  a 
basic  quality  rating,  evaluators  should  decide  whether  the 
behaviors  observed  fall  into  the  low  end  of  the  basic  quality 
range  (values  1  or  2),  the  middle  of  the  range  (values  3,  4,  or 
5),  or  the  high  end  of  the  range  (values  6  or  7) .  Once  the 
general  range  of  response  is  selected,  use  the  anchors  to  help 
select _ the  final  rating  value.  Cautions  included  in  the 
guidelines  emphasized  that  evaluators  should  use  the  behavioral 
anchors  as  the  standard  for  making  ratings  and  avoid  lapsing  into 
rating  crews  in  comparison  to  one  another. 
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Figure  4.  Aircrew  coordination  Basic  Qualities  and  their  relationships 
to  ATM  Aircrew  Coordination  Elements  and  Crew  Coordination  Objectives. 


Working  Group  members  were  concerned  about  the  ability  of 
instructor-evaluators  to  effectively  interpolate  and  apply  the 
behavioral  anchors  to  determine  ratings.  As  a  result,  a  more 
structured  method  of  decomposing  and  scoring  the  Basic  Qualities 
was  provided  to  evaluators  for  use  as  an  alternative  to  the 
behavioral  anchors.  The  Crew  Coordination  Debriefing  and  Rating 
Guide  (Optional)  decomposed  each  Basic  Quality  into  a  set  of 
issues  for  the  evaluator  to  focus  on  during  the  mission.  For 
each  issue,  the  guide  posed  a  series  of  questions  regarding 
different  aspects  of  observable  performance  along  with  three 
general  answers  to  each  question  reflecting  the  levels  of 
performance:  superior,  acceptable,  and  very  poor.  The  method 

permitted  evaluators  to  compute  an  average  score  for  each  issue 
associated  with  a  Basic  Quality  and  an  overall  score  for  the 
Basic  Quality  itself. 

The  Basic  Qualities  and  behavioral  anchors,  rating  scale, 
and  rating  guidelines  are  shown  in  Appendix  B. 

Note:  The  discussion  of  major  activities  to  develop  the 
testbed  evaluation  package  transitions  at  this  point  from  general 
topics  supported  by  aviation-wide  examples  to  more  specific 
topics  illustrated  by  UH-60  unique  examples  used  in  the  testbed 
itself.  The  more  specific  topics,  like  ATM  tasks,  address  the 
details  necessary  to  implement  the  crew  coordination  evaluation 
method  for  aircrews  flying  any  Army  aircraft.  For  example,  the 
discussion  on  ATM  tasks  and  grade  slips  applies  to  all  aircraft, 
but  only  the  UH-60  rated  crew  member  grade  slip  developed  for  the 
testbed  is  shown. 

ATM  Tasks  and  Grade  Slips 

The  testbed  evaluation  package  included  crew  performance  of 
aircraft  specific  tasks.  The  new  ATMs,  with  crew  coordination 
duties  specified,  served  as  the  source  of  tasks  to  include  in  the 
validation  testbed.  The  AH-64  and  UH-60  draft  ATMs  were  reviewed 
to  select  crew  coordination  intensive  tasks  that  could  be 
included  in  a  tactical  scenario  for  evaluation  in  a  visual  flight 
simulator.  Tasks  like  navigation,  target  acquisition  and 
engagement,  threat  evasion,  external  loads,  and  instrument  flight 
were  selected  and  reviewed  by  instructor  pilots  and  simulator 
operators  for  suitability  to  evaluate  crew  coordination. 

Initial  guidance  was  to  develop  crew  coordination  training 
and  evaluation  materials  that  implemented  the  new  ATMs . _  Project 
staff  met  with  the  crew  coordination  Working  Group  to  discuss  the 
adequacy  of  the  new  ATM  grade  slips  and  grading  procedures  to 
evaluate  crew  coordination  performance.  The  Working  Group 
specified  that  the  Maneuver/ Procedure  Grade  Slip  for  each 
aircraft  be  used  to  evaluate  ATM  tasks  but  agreed  to  consider 
modifications  if  needed  to  support  the  use  of  the  ACE  Checklist 
Basic  Qualities  and  behaviorally  anchored  rating  system  (BARS) 
measures . 
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Project  staff  analyzed  the  new  ATMs  and  developed  a  set  of 
modifications  to  the  grade  slips  and  grading  system  for  Working 
Group  approval.  Modifications  were  developed  based  on  the  AH-64 
ATM  and  then  applied  to  the  UH-60  aircraft  for  validation  in  the 
crew  coordination  testbed.  The  modifications  were  developed  to 
be  applicable  to  all  Army  aircraft. 

The  Maneuver/ Procedure  Grade  Slip  was  modified  to  provide 
more  space  for  crewmember  information,  multiple  entries  in  the 
grade  block  for  each  task,  and  a  look-up  table  of  crew 
coordination  Basic  Qualities  at  the  bottom  of  each  page.  A  grade 
block  was  provided  to  enter  a  summary  rating  (BARS  scale  numbers 
1,  2,  ...7)  for  each  Basic  Quality  at  the  bottom  of  the  last 
page.  Minor  modifications  to  the  instructions  for  use  of  the 
Battle-Rostered  Crew  Evaluation/Training  Grade  Slip,  introduced 
in  the  new  ATMs,  were  developed  to  accommodate  crew  coordination 
grading  requirements.  The  modified  instructions  substituted  the 
Aircrew  Coordination  Training  Grade  Slip  for  the  list  of  crew 
tasks  and  applied  an  expanded  grading  system,  described  below,  to 
the  overall  flight  grade.  Use  of  this  grade  slip  in  conjunction 
with  the  Aircrew  Coordination  Training  Grade  Slip  provided  a  link 
between  initial  and/or  refresher  crew  coordination  training  and 
the  unit's  continuation  training. 

Modifications  included  an  expanded  grading  system  for  crew 
coordination  training  that  includes  both  the  technical  and  crew 
coordination  aspects  of  each  ATM  task.  The  modification  expands 
the  current  satisfactory  (S)  or  unsatisfactory  (U)  grading  system 
to  provide  more  precise  evaluation  of  crew  coordination  strengths 
and  weaknesses  by  breaking  the  satisfactory  grade  into 
satisfactory  plus  (5+) ,  satisfacto^  (S) ,  and  satisfactory  minus 
(S-)  grades.  Written  guidance  to  implement  the  expanded  grading 
system  includes  specifics  to  identify  the  crew  coordination  Basic 
Qualities  (numbers  1,  2,  ...13),  if  any,  that  contributed  to  less 

than  satisfactory  ATM  task  performance.  The  grading  guidelines 
provided  instructions  to  make  a  written  comment  for  tasks  graded 
S+  or  U,  to  make  a  summary  rating  for  each  Basic  Quality,  and  to 
grade  the  overall  flight. 

The  Working  Group  approved  the  modifications  for  evaluating 
crew  coordination  but  restricted  the  Aircrew  Coordination 
Training  Grade  Slip  to  initial  and  refresher  crew  coordination 
training  only.  Instructions  on  the  grading  system  and 
reproducible  blanks  of  the  final  Aircrew  Coordination  Training 
Grade  Slip  (See  Figure  5)  were  included  in  the  Crew  Coordination 
Exportable  Training  Package  (Pawlik,  Simon,  Grubb,  &  Zeller, 

1992)  . 

Scenario  Guidance 

Discussions  with  the  crew  coordination  Working  Group  and 
detailed  reviews  of  current  Army  publications,  including  the  new 
ATMs,  emphasized  the  importance  of  developing  realistic  tactical 
situations  to  accurately  assess  crew  performance. 
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AIRCREW  COORDINATION  TRAININO  GRADE  SUP 


.Figure  5.  Aircrew  coordination  training  grade  slip  (page  1  of  2) 
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PAGE  2.  AIRCREW  COORDINATION  TRAINING  GRADE  SLIP 


Figure  5.  Aircrew  coordination  training  grade  slip 
(page  2  of  2 ) . 
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Project  staff  researched  the  adequacy  of  existing  preplanned 
scenarios  maintained  by  flight  simulator  facilities  and  the 
published  guidance  on  developing  scenarios.  The  diverse  missions 
and  level  of  detail  in  the  preplanned  scenarios  made  them 
unusable  for  the  testbed  without  major  modifications.  Guidance 
on  scenario  development  published  in  doctrine  and  training 
literature  was  too  broad  to  apply  without  interpretation.  The 
need  to  provide  guidance  on  how  to  develop  tactical  scenarios  was 
confirmed  by  visits  to  units  in  the  field.  As  a  result,  two 
tactical  scenarios  for  UH-60  equipped  units  were  prepared  for 
the  validation  testbed  evaluation  package. 

The  scenarios  were  methodically  developed  to  realistically 
illustrate  the  primary  missions,  conditions,  and  situations 
needed  to  evaluate  crew  coordination  mission  performance  areas 
and  tasks.  Detailed  outlines  to  identify  mission  segments, 
tactical  events,  and  ATM  tasks  were  prepared  to  document  the 
scenario  development-evaluation  process.  Tactical  realism  was 
ensured  by  preparing  detailed  air  mission  briefings  for  the 
crews,  narrative  scripts  for  simulator  operators,  instructions  to 
assist  simulator  operators  program  the  scenario,  and  written 
guidance  to  assist  crew  coordination  evaluators.  The 
requirements  considered,  steps  taken,  information  sources 
accessed,  and  formats  that  the  project  staff  used  to  document  the 
validation  testbed  scenarios  constitute  the  scenario  guidance 
materials  for  evaluating  crew  coordination.  These  materials  are 
also  included  in  the  Crew  Coordination  Exportable  Training 
Package  (Pawlik,  Simon,  Grubb,  &  Zeller,  1992) . 

Both  general  and  specific  requirements  were  developed  for 
scenarios  and  situational  training  exercises  to  be  used  in  flight 
simulators  and  aircraft,  respectively.  The  following  general 
requirements  to  support  crew  coordination  evaluation  were 
reviewed  and  approved  by  the  Working  Group: 

1.  Focus  on  unit's  mission  essential  task  list  (METL) . 

2 .  Be  consistent  with  the  guidance  for  crew  training 
contained  in  TC  1-210,  Aircrew  Training  Program,  Commander's 
Guide  to  Individual  and  Crew  Training  Program,  and  the 
appropriate  ATM. 

3.  Involve  battle-rostered  crews.  Note:  Guidelines  in  TC 
1-210  establish  the  requirement  for  commanders  to  battle  roster 
crews.  The  USAAVNC  prescribes  that  battle  rostered  crews  fly 
together  at  least  once  eve^  180  days.  The  USAAVNC  guidance 
assumes  that  battle  rostering  promotes  the  development  and 
maintenance  of  effective  crew  coordination.  However,  crews 
participating  in  the  validation  testbed  had  not  yet  been  battle 
rostered  by  their  unit. 

4.  Emphasize  crew  tasks  developed  as  a  part  of  the  unit's 
collective  training. 
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Specific  requirements  were  included  to  establish  the  amount 
of  time,  types  of  events,  degree  of  difficulty,  desirability  of 
video  recording,  and  evaluator  duties  for  crew  coordination 
evaluation  scenarios. 

Materials  were  prepared  to  document  the  suggested  steps  and 
products  from  each  step  in  developing  a  crew  coordination 
evaluation  scenario. 

1.  Select  a  common  tactical  mission(s).  Materials  include 
Table  4,  Missions  of  Army  Aviation  Aircraft. 

2.  Identify  unit  mission  essential  tasks. 

3.  Incorporate  activities  and  ATM  tasks  that  emphasize 
crew  coordination. 

4.  Develop  a  scenario  outline  (See  Table  5) . 

5.  Identify  activity  breakpoints  and  describe  the  major 
activities  and  focus  for  each  scenario  segment  (See  Table  6) . 

6.  Select  ATM  tasks. 

7.  Transpose  scenario  outline  to  the  tactical  training 
area  available. 

8.  Develop  an  OPORD  and/or  an  air  mission  briefing  to 
include  a  simulator  operator  script  for  simulator  scenarios. 

A  checklist  of  scenario-based  materials  was  developed  as  a 
guide  for  evaluators  to  instill  a  sense  of  unit  mission  and 
mission  planning  realism  to  crew  coordination  evaluations  (see 
Table  7) .  Visits  to  units  in  the  field  provided  a  forum  for 
review  of  the  scenario  development  guidance  materials  for  their 
suitability  to  evaluate  crew  coordination. 

Mission  Performance  Measures 

The  new  ATMs  emphasized  that  research  has  shown  direct, 
positive  effects  of  crew  coordination  on  flight  safety  and 
mission  performance.  For  this  reason,  actions  were  taken  to 
include  measures  of  mission  performance  in  the  crew  coordination 
validation  testbed  design.  Thus,  a  set  of  mission  performance 
measures  was  developed  to  extend  the  evaluation  of  crew 
performance  of  ATM  tasks  to  overall  mission  performance. 

Measures  of  mission  performance  captured  via  data 
extrapolated  from  video  tapes  of  crew  missions  in  a  flight 
simulator  and  reported  in  previous  research  (Simon,  1991)  showed 
that  a  positive  correlation  exists  between  crew  coordination  and 
mission  performance.  The  mission  performance  measures  used  in 
the  1990  experiment  were  navigation,  threat  encounters,  and 
instrument  approach  procedures . 
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Table  4 


Missions  of  Army  Aviation  Aircraft^ 


Mission 


Roies 


Aircraf t 


Observation 

Observation  helicopters  perform  visual 
observation  and  target  acquisition. 


Attack 

The  primary  mission  of  attack 
helicopters  is  to  destroy  enemy 
armored,  mechanized,  and  helicopter 
forces. 


Utility 


Rotary  wing 

Utility  helicopters  perform  a  variety 
of  missions  to  include  air  assault, 
air  movement,  command  and  control,  and 
MEDEVAC  operations. 


Rf^connaissance  (route,  area, 
zone) 

Security  (cover,  guard, 
screen) 

Command,  control, 
communications,  and 
intelligence  enhancement 
Aerial  adjustment  of  field 
artillery 
Surveillance 
NBC  reconnaissance 
Laser  designation  for 
.  precision  guided  munitions 
(OH-58D  only) 

Antiarmor 
Antipersonnel 
Air  combat 

Suppression  of  enemy  air 
defenses 

Joint  air  attack  team 
operations 

-  '  Joint  second  echelon  attack 
Antimateriel 
Laser  designation  for 
precision  guided  munitions 
(AH-64  only) 


0H-6A, 

OH  -  ‘‘.MA, 
OH-58C, 
0H-5SD,  and 
RAH-66 


A^64, 

RAH-66, 

AH-iE, 

AH-IS, 

AH-iP, 

AH-IF,  and 

UH-iM 


Air  assault  and  combat  assault 
of  combat  forces 
Air  movement  of  supi:)lies, 
equipment,  and  personnel 
Aerial  evacuation  of 
equipment,  casualties,  and 
prisoners  of  war 
Aerial  delivery  of  scatterable 
mines  and  sensors 
Combat  search  and  rescue 
Command,  control, 
communications,  and 
intelligence  enhancement 


UH-IH, 
UH-IV 
(MKUEVAC) , 
and  UH-6UA 


Fixed  wing 

Utility  fixed-wing  aircraft  are 
employed  to  move  personnel  and 
equipment  and  to  support  commanders 
and  their  staffs. 


Command ,  cont  rol , 
communications,  and 
intelligence  enhancement 
Administration 
Liaison 

Ap  r  «;)mpd  i  ca  1  evacua  t  i  on 


U-2iA, 
C-12A/C, 
UV-i8A,  and 
C-20 


Cargo 

Cargo  helicopters  perform  a  variety  of 
missions  from  air  movement  of  combat 
power  to  air  movement  of  troops  and 
cargo. 


Air-move  combined  arms  forces 
and  equipment 

Emplace  fipld  artillety  anr) 
of  ijer  file  suppt.nl  anupl  :: 
Reposition  tactical  air 
defense  weapons  and  systems 
Perform  medical  evacuation 
Move  combat  power,  troops, 
logistical  supplies,  and 
equipment  forward,  laterally, 
and  rearward 

Perform  logistics  over-the- 
shore  operations 
Air-move  conventional, 
nuclear,  and  chemical 
munitions 


CH-47A, 
r'U-47B, 
r'll  *47r\ 

(V  4/1), 
CH-54A,  and 
CH-54B 
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Table  4 


Missions  of  Army  Aviation  Aircraft®  (Cont.) 


Mission 


Roles 


Aircraft 


Special  electronic  mission  aircraft 

Special  electronic  mission  aircraft 
perform  a  variety  of  intelligence  and 
electronic  warfare  operations. 


Special  Operations  Aviation 

SOA  aircraft  perform  a  variety  of 
missions  to  support  special  operations 
forces. 


Process  and  relay  high  value 
int#^l  1  igence  information  to 
maneuver  commanders 
Provide  communications 
intelligence 

Provide  electronic  collection 
Collect,  process,  and  analyz*^ 
infrai^^d,  radar,  and  iiihottigrapliic 
imagery 

Support  deception  operations  to 
deceive  and  deny  critical  combat 
information  to  enemy  forces 
Provide  direction  finding, 
interception,  and  jamming  of 
communications  emitters 


EH-1, 

EH-60, 

RV/OV-1,  RU- 
21,  and  RC- 
12 


Clandestinely  penetrate  denied  MH-6,  AH-6, 

enemy  areas  MH-60K,  and 

Assault,  resupply,  insert,  or  MH-47E 

extract  SOF 

Conduct  aerial  security, 
reconnaissance,  surveillance,  and 
electronic  warfare  support  of 
special  operations  missions 
Provide  airborn<^  command,  control, 
and  communications  enhancement 
Support  coordinated  and 
synchronized  joint,  combined,  or 
host-nation  special  operations 
Perform  aircraft  strategic  s^lf- 
deploymenC  operations 
F’erform  limited  aer(:)medicc»l 
evacuation 

Perform  search  and  rescue 
operations 

Conduct  aerial  mine  delivery 
operations 

Perform  general  aviation  support 
missions,  as  necessary 


^Source:  FM  1-100,  Doctrinal  Principles  for  Army  Aviation  in  Combat  Operations, 

February  1989 


28 


Table  5 

Air  Assault  &  Air  Movement  Scenario  Outline 


Segment 


Performance  measures 


ATM  tasks 


1.  Pre-mission 
Planning 


Mission  Planning  &  Rehearsal 
Mission  brief ing/brief-back 


2078 -terrain  fit  msn  plan 
1004-PPC 

1000-Mission  brief 


2.  AA  to  PZ 


Material  malfunction  (major) 


3.  PZ  to  LZ  to  PZ  Navigation  (corridors) 

Time  to  fly  segment 
Time  of  arrival 
Threat  avoidance  U  evasion 
Material  malfunction  (minor) 


4.  PZ  to  LZ  to  PZ  Navigation  (corridors) 

Time  to  fly  segment 
Threat  avoidance  &  evasion 


5.  PZ  to  AA  Inadvertant  IMC 

Instrument  recovery 


1007 -Start /run-up 
101 6 -Hover  pwr  ck 
1018-VMC  takeoff 
1026-Elect-aided  nav 
2079-Terrain  fit  nav 
2081-Terrain  fit 
1023 -Fuel  mgt  proced 
1068-Emergency 
1095-Operate  ASE 

2008- Evasive  maneuvers 
1028-VMC  approach 

1016-Hover  pwr  ck 
1018-VMC  takeoff 

2009- Multiaircraft  opns 
1026-Elect-aided  nav 

2 07 9 -Ter rain  fit  nav 
2081-Terrain  fit 
1023-Fuel  mgt  proced 
1095-Operate  ASE 
2008-Evasive  maneuvers 
1028 -VMC  approach 

2016-External  load  opns 
1016-Hover  pwr  ck 
1018-VMC  takeoff 
1026-Elect-aided  nav 
2079-Terrain  fit  nav 
2081-Terrain  fit 
1023-Fuel  mgt  proced 
1095-Operate  ASE 
2008-Evasive  maneuvers 
1028-VMC  approach 

1018-VMC  takeoff 
1026-Elect-aided  nav 
2079-Terrain  fit  nav 
2081-Terrain  f 1 t 
1023-Fuel  mgt  proced 
1095-Operate  ASE 
2008-Evasive  maneuvers 
1083-VHIRP 
3  07 6 -Rad  in  nav 
1081"  N()n  pL<‘c  i  Lon  ap[n. 
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Table  6 


Scenario  Segment  Information 


Segment  1:  Premission  planning 

Description:  The  premission  planning  segment  begins  when  the  crew  receives  the 

mission  briefing  and  includes  all  preparatory  tasks  associated  with  planning  the 
tactical  mission.  These  tasks  include  terrain  flight  mission  planning,  performance 
planning,  assigning  crewmember  responsibilities,  and  all  required  briefings  and  brief- 
backs.  The  segment  ends  when  the  crew  completes  all  required  briefings  and  prepares  to 
enter  the  simulator. 


Segment  2:  Movement  from  the  assembly  area  (AA)  to  the  initial  pick-up  zone  (PZ) 

Description:  The  segment  begins  when  tlie  c^rew  enters  the  simulator  and  veLifi<-*s  that- 

initial  start  and  run-up  procedures  are  complete.  During  this  segment,  the  crew 
repositions  the  aircraft  from  the  AA  to  the  initial  PZ  in  preparation  for  an  ait- 
assault  mission.  The  segment  includes  an  emergency  caused  by  an  aircraft  system 
malfunction  which  should  result  in  a  precautionary  landing  in  the  PZ.  The  segment  ends 
when  the  crew  completes  the  precautionary  landing. 


Segment  3:  Cross-FLOT  air  assault  mission 

Description:  The  segment  begins  when  the  troops  have  been  loaded  on  the  aircraft.  It 
involves  moving  troops  along  a  prescribed  route  in  a  medium- to- h igh  threat 
environment,  delivering  them  to  the  LZ,  and  then  returning  to  the  PZ .  The  crew  will 
act  as  flight  lead  for  a  flight  of  5  UH-60  helicopters  with  no  changes  in  lead  or 
formation.  The  crew  must  accurately  navigate  within  prescribed  corridors  while 
avoiding  and  evading  threat  to  deliver  the  troops  to  the  correct  location  at  the 
correct  time.  The  segment  includes  a  minor  malfunction  which  will  be  removr-d  as  soon 
as  the  crew  detects  and  verbally  recognizes  the  malfunction.  The  segment  ends  when  the 
crew  returns  to  the  PZ. 


Segment  4:  External  load  air  movement  mission 

Description:  The  segment  begins  when  the  crew  takes  off  to  pick  up  the  external  load. 

It  involves  moving  an  external  load  along  a  prescribed  route  in  a  medium- to-high 
threat  environment  to  resupply  a  friendly  unit  located  near  the  forward  line  of  troops 
(FLOT) .  The  crew  must  accurately  navigate  within  prescribed  corridors  while  avoiding 
and  evading  threat  to  deliver  the  external  load  to  the  correct  location.  The  crew 
then  returns  to  the  PZ  in  preparation  for  a  follow-on  mission.  The  segment  ends  when 
the  aircraft  returns  to  the  PZ. 


Segment  5:  Movement  from  the  PZ  to  the  assembly  area 

Description:  The  segment  begins  when  the  aircraft  departs  the  PZ  enroute  back  to  the 

AA.  During  the  flight,  the  crew  encounters  an  inadvertent  entry  into  instrument 
meteorological  conditions  (IMC)  .  The  crew  mu.st  then  plan  and  exet^ute  a  iioni na j  s  i  on 
instrument  approach  to  transition  back  to  visual  meteorological  coiiditions  (VMC)  ,  Tin* 
segment  ends  when  the  crew  completes  a  safe  landing. 
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Table  7 


Scenario  Materials 


Scenario  item 

student 

Instructor 

simulator 

operator 

OPORD  and/or  air 
mission  briefing* 

X 

X 

X 

Scenario  outline 

X 

X 

Scenario  segment  info 

X 

X 

Tactical  map 

X 

X 

X 

Approach  plate 

X 

X 

X 

Grade  slip 

X 

‘Provide  an  instructor  operator  script  for  simulator 
scenarios • 


flight  simulator.  After  discussions  with  simulator  operators  and 
instructor  pilots,  the  mission  performance  measurement  areas 
listed  below  were  selected  to  evaluate  crew  coordination  in  the 
validation  testbed. 

1.  Terrain  flight  navigation.  Scenario  mission 
requirements  demanded  close  compliance  with  specific  flight 
routes  and  schedules. 

2.  Threat  avoidance  and  evasion.  Enemy  situation  in  the 
scenario  included  different  enemy  anti-aircraft  systems. 

3.  Aircraft  emergencies.  Scenario-related  aircraft 
malfunctions  were  programmed  to  occur  during  the  mission. 

4.  Unexpected  event.  Visibility  and  weather  conditions, 
forecast  to  deteriorate  throughout  the  mission,  were  adjusted  to 
create  inadvertent  instrument  meteorological  conditions. 

5.  Instrument  flight  recovery.  Aircraft  and  landing  site 
equipment  availability  were  reduced  to  require  a  non-precision 
instrument  approach  procedure. 

6.  Mission  threatening  crew  error.  Potential  accident 
and/or  injury  situations  were  presented  throughout  the  scenario. 

Doctrinal,  training,  and  equipment  publications  were  used  to 
develop  specific  performance  measures  within  each  mission  area. 
Written  descriptions  were  prepared  to  detail  what  to  measure,  how 
to  collect  measurement  data,  and  parameters  or  metrics  for  each 
performance  measure.  For  example,  five  specific  performance 
measures  were  developed  for  Terrain  Flight  Navigation  (see  Table 
8) .  Expanded  descriptions  of  the  data  elements  for  each  metric 
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Table  8 


Terrain  Flight  Navigation  Performance  Measures 


What  to  measure 

How  to  collect 

Measurement 

parameters 

Number  (N)  of 
deviations  from  the 
corridor  due  to 
misorientatio 

FS  printout,  page 

25,  "Cross  country 
map"  (12  X  12  K  or 

24  X  24  K)  with 
ground  track  trace; 
verified  by  video 
tape  review 

Sum  (N) 

Distance  of 
deviation  outside  of 
corridor  due  to 
misorientation 

FS  printout,  page 

25,  "Cross  country 
map"  (12  X  12  K  or 

24  X  24  K)  with 
ground  track  trace; 
verified  by  video 
tape  review 

Sum  (N) 

Sum  (N)  <500m 

Sum  (N)  >500m  <1500m 
Sum  (N)  >1500m 

Deviation  (seconds) 
from  required  time 
of  arrival  at 
landing  zone 

Live 

observation/time  on 
tape 

Actual  time  compared 
to  time  designated 
in  OPORD/FRAG 

Number  (N)  of 
mission  (route) 
segments  completed 

Live  observation; 

FS  printout 

Sum  (N) 

Time  (sec)  to  fly 
each  mission  segment 

Live 

observation/time  on 
tape 

Actual  time  compared 
to  time  designated 
in  OPORD/FRAG 

Analysis  of  the  Mission  Training  Plan  (MTP)  missions  for 
attack  and  utility  helicopter  units  guided  the  identification  of 
mission  performance  areas  and  helped  to  define  specific  mission 
performance  measures  for  evaluation.  Potential  mission 
performance  areas  were  studied  to  ensure  that  they  could  be 
realistically  included  in  scenario  segments  and  evaluated  in  a 
were  prepared  to  support  the  testbed  data  collection  and  analysis 
efforts . 
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Instructor  pilot-evaluators  reviewed  these  measures  for 
suitability  as  they  evaluated  their  crews  and  collected  data  on 
measures  like  aircraft  emergencies  and  unexpected  events  during 
the  crew  coordination  testbed.  More  complete  written 
descriptions  of  the  crew  coordination  mission  performance 
measures  used  in  the  testbed  are  at  Appendix  D. 

Evaluator  Workbook 

The  addition  of  crew  coordination  actions  to  ATM  task 
standards  placed  an  increased  demand  on  individual  evaluator's 
full  attention.  Data  collection  techniques  for  the  crew 
coordination  validation  testbed  required  that  instructor  pilot- 
evaluators  augment  the  project  staff's  direct  observation  and 
collection  of  specific  mission  performance  measures.  A  means  was 
needed  to  allow  evaluators  to  collect  mission  performance  data 
without  diverting  their  attention  from  the  primary  task  of 
evaluating  crew  performance. 

An  Evaluator  Workbook  was  developed  to  organize  the 
evaluator's  questionnaire,  grade  slips,  ACE  Checklist  behavioral 
measures  and  other  materials  in  the  same  order  that  they  would  be 
refered  to  during  the  evaluation  process.  Previous  experience 
with  similar  data  collection  requirements  guided  the  development 
of  a  worksheet  to  facilitate  rapid  and  accurate  recording  of 
evaluator  entries.  Evaluator  worksheets  were  developed  to  permit 
circle  or  fill-in  type  entries  for  crew  coordination  performance 
and  data  collection  items. 

Organized  by  mission  segments,  the  evaluator  worksheets  were 
developed  in  accordance  with  the  outline  for  each  scenario.  A 
separate  text  block  was  used  for  each  worksheet  topic  to  help 
evaluators  scan  and  locate  desired  information  quickly.  Topic 
blocks  were  arranged  in  scenario  outline  sequence  (see  Table  9) . 

1.  Segment  number.  This  block  includes  the  segment  title 
and  a  description  of  the  events  that  begin  and  end  the  segment, 
including  mission  performance  measure  related  events. 

2.  ATM  task  block.  The  ATM  task  block  provides  evaluators 
with  preprinted  grade  and  rating  options  for  each  task.  Blank 
space  is  provided  for  evaluator  notes.  Circle  or  fill-in 
responses  for  data  collection  events  are  printed  in  bold 
characters . 

3.  Segment  overall.  The  last  block  in  a  mission  segment, 
this  topic  presents  criteria  to  evaluate  the  crew's  performance 
of  this  segment  as  if  it  were  a  separate  mission. 

4.  Basic  Qualities  reference.  Located  at  the  bottom  of 
each  worksheet  page,  this  block  displays  the  Basic  Quality  short 
titles  for  reference. 
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Table  9 


Evaluator  Worksheet 


SEGMENT  5:  Movement  from  the  PZ  to  the  assembly  area. 

DESCRIPTION:  The  segment  begins  when  the  aircraft  departs  the  PZ  enroute  back  to  the  AA.  During  the 
flight,  the  crew  encounters  an  inadvertent  entry  into  instrument  meteorological  conditions  (IMC).  The  crew 
must  then  plan  and  execute  a  nonprecision  instrument  approach  to  transition  back  to  visual  meteorological 
conditions  (VMC).  The  segment  ends  when  the  crew  completes  a  safe  landing. 


TASK  1018  Perform  VMC  takeoff 

GRADE:  S+  S  S-  U  Basic  Qualities: 

NOTES; 


TASK  1026  Perform  electronic-aided  navigation 

GRADE:  S+  S  S-  U  Basic  Qualities: 

NOTES: 


TASK  2079  Perform  terrain  flight  navigation 

GRADE:  S+  S  S-  U  Basic  Qualities: 

NOTES: 


AIRCREW  COOflONATION  BASIC  QUALITIES 


1 

2. 

3 

4. 

S 

t. 

7 

I. 

WMMM 

10. 

n. 

12. 

13. 

CREW 

PLAN 

DECI- 

WORK 

UNEXP 

NFO 

SIT 

COMM 

CROSS 

NFO 

ADVOC/ 

AAR 

CLI¬ 

MATE 

RE¬ 

HEARSE 

SION 

TECH 

LOAD 

EVENTS 

XFER 

AWARE 

ACK 

MONI¬ 

TOR 

OF¬ 

FERED 

ASSERT 
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Table  9 


Evaluator  Worksheet  (Continued) 

SEGMENT  5;  (Concluded) _ 

TASK  1083  Perform  inadvertent  IMG  procedures/VHIRP 

GRADE:  S+  S  S-  U  Basic  Qualities: 

NOTES: 

-  VHIRP  steps  accomplished?  _ of  5 


TASK  1076  Perform  radio  navigation 

GRADE:  S+  S  S-  U  Basic  Qualities: 

NOTES: 


-  Number  of  heading  deviations  >  +/-  5  degrees? 

-  Number  of  altitude  deviations  >  -f/- 100  feet? 


TASK  1081  Perform  nonprecision  approach 

GRADE:  S+  S  S-  U  Basic  Qualities: 

NOTES: 

•  Inbound  leg  of  approach  property  timed?  Yes/No  (Circle  one) 


SEGMENT  5  Crew's  ability  to  cope  with  inadvertent  entry  into  instrument 

OVERALL  meteorological  conditions  and  their  ability  to  plan  and  execute  a  non-precision  instrument 
approach. 

GRADE:  S+  S  S-  U  Basic  Qualities:  _ , _ (OPTIONAL) 

NOTES: 


AIRCREW  COORONATiON  BASIC  QUALITIES 


1. 

2. 

3. 

4. 

HHI 

9. 

■□■1 

10. 

11. 

12. 

13. 

CREW 

PLAN 

DECI- 

WORK 

NFO 

COMM 

CROSS 

NFO 

ADVOC/ 

AAR 

CLI¬ 

RE¬ 

SION 

LOAD 

XFER 

ACK 

MONI¬ 

OF¬ 

ASSERT 

MATE 

HEARSE 

TECH 

inai 

■iifl 

■■I 

TOR 

FERED 
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Instructor  pilot-evaluators  used  the  evaluator  worksheets 
during  the  validation  testbed  to  record  information  for  grade 
slip,  crew  debriefing,  and  mission  performance  data  collection 
purposes.  The  Evaluator  Workbook  was  a  prototype  of  the  Crew 
Coordination  Exportable  Evaluation  Package  described  later  in 
this  report. 

Interaction  with  Training  Course  Development 

The  Army's  Crew  Coordination  Program  for  aviation  depends 
largely  on  instructor  pilots  who  train  their  crews  in  field 
units.  Course  development  discussions  emphasized  the  need  to 
apply  classroom  instruction  to  the  tasks  that  crews  perform  in  a 
typical  mission  environment.  As  a  result,  two  training 
evaluation  flights  in  a  visual  simulator  or  an  aircraft  were 
designed  into  the  validation  testbed  training  course. 
Additionally,  two  evaluation  flights,  one  pre-training  and  one 
post-training,  were  included  in  the  validation  testbed  schedule. 
The  post-training  evaluation  flight  was  considered  the  course 
completion  examination. 

Army  Research  Institute  (ARI)  emphasized  the  need  for  the 
training  and  evaluation  to  be  conducted  within  identical 
frameworks.  Direct  participation  in  the  design  and  development 
of  the  crew  coordination  training  package  ensured  that  the 
evaluation  process  supported  the  training  strategy  and  addressed 
the  specific  crew  coordination  learning  objectives.  See  the 
Development  of  Candidate  Crew  Coordination  Training  Methods  and 
Materials  technical  report  for  details. 

Project  staff  proceeded  to  develop  instructional  materials, 
case  study  examples,  and  practice  exercises  to  teach  instructor 
pilot-evaluators  and  unit  trainers  about  the  evaluation  method 
and  how  to  use  the  evaluation  materials.  Specific  sections 
provided  the  following  instruction  to  crew  coordination 
evaluators : 

1.  Evaluation  Procedures  and  Scenario  Development. 
Procedures  for  assessing  crew  coordination  performance  and 
guidelines  for  developing  scenarios. 

2.  Aircrew  Coordination  Training  Grade  Slips.  Describes 
the  grade  slips  and  the  expanded  grading  system  for  aircrew 
coordination  training. 

3.  Aircrew  Coordination  Evaluation  Workshop  Exercises. 
Provides  evaluators  with  classroom  exercises  to  recognize  and 
evaluate  crew  coordination  performance. 

4.  Aircrew  Coordination  Evaluation  Process.  Provides  a 
summary  of  evaluation  actions  to  include  video  recording  options. 

5.  Scenario  Guidance.  Provides  specific  guidance  and 
examples  to  develop  scenarios  for  visual  flight  simulators  and 
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situational  training  exercises  for  crew  coordination  evaluations 
conducted  in  aircraft. 

6.  Scenario  Familiarization  and  Evaluation.  Familiarizes 
evaluators  with  a  complete  simulator  scenario  and  provides 
experience  evaluating  crews  in  the  simulator  or  aircraft. 

Course  materials  used  to  train  evaluators  were  rehearsed 
with  the  crew  coordination  Working  Group  to  ensure  clarity  and 
completeness  of  the  evaluation  process  and  to  gain  their 
approval . 

Exit  Interviews 

Participant  debriefings  were  considered  an  integral  part  of 
the  validation  testbed  evaluation  package.  Two  exit  interviews 
were  constructed.  One  exit  interview  was  constructed  for  the 
participating  aircrews  and  the  other  was  for  the  participating 
instructor  pilots  and  unit  trainers.  Project  staff  drafted  the 
initial  exit  interviews  based  on  questions  previously  used  in  the 
interviews  conducted  during  the  1990  crew  coordination 
experiment .  Both  interviews  were  reviewed  and  approved  by  the 
crew  coordination  Working  Group. 

The  general  sections  in  the  crewmember  exit  interview  asked 
for  feedback  on  administrative  aspects  of  the  course,  the 
structure  and  presentation  of  the  course  of  instruction,  the 
flight  simulator  portion  of  the  training,  and  general 
observations  regarding  the  adequacy  and  worthiness  of  the 
training  program.  The  evaluators  and  trainers  were  asked  for 
feedback  about  the  adequacy  of  the  course  of  instruction  that 
they  received  and  then  used  to  teach  the  aviators,  the  scenarios 
used  during  the  evaluation  phase,  the  evaluator's  workbook,  their 
use  of  the  Basic  Qualities  and  the  modified  grade  slips,  and 
general  observations  regarding  the  training  and  evaluation 
process  used  in  the  validation  testbed. 

Both  interviews  were  designed  to  be  group  administered; 
however,  instructors  were  also  interviewed  one-on-one  and  then 
later  as  a  group.  To  expedite  the  interview  process, 
participants  were  given  the  interview  questions  several  days 
prior  to  the  scheduled  interview.  For  each  of  the  five  two-hour 
group-administered  interviews,  eight  participants  composed  of 
four  aircrews  or  the  instructor  cadre  took  part  in  any  one 
session. 

The  exit  interviews,  "Aircrew  Coordination  Training 
Validation  Testbed,  Crewmember  Exit  Interview, "  and  "Aircrew 
Coordination  Training  Validation  Testbed,  Evaluator  and  Trainer 
Exit  Interview"  are  presented  in  Appendix  E. 
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Summary  of  Validation  Testbed  Activities  and  Lessons  Learned 

Condensed  into  a  four-week  period  from  3  August  to  4 
September  1993,  the  validation  testbed  conducted  at  Fort 
Campbell,  Kentucky  by  aviators  from  the  101st  Airborne  (Air 
Assault)  Division,  adequately  stressed  the  evaluation  package. 

Two  utility  helicopter  tactical  missions  were  used  for  the 
crew-level  evaluations  and  to  assess  changes  in  crew  mission 
performance.  The  baseline  evaluation  was  conducted  prior  to  the 
crew  coordination  training  {pretraining  condition) .  The  second 
evaluation  was  administered  after  the  training  (post-training 
condition) .  The  two  missions  were  very  similar  in  difficulty  in 
terms  of  time  stress,  navigational  demands,  quantity  and 
capabilities  of  simulated  threat,  etc.  The  objectives  and  tasks 
incorporated  into  the  scenario  were  all  made  to  present  two 
equally  difficult  missions  to  the  aircrews.  Four  IPs  and  four 
UTs  received  training  to  present  the  Aircrew  Coordination  Student 
Course  and  to  perform  the  pre-  and  post -training  evaluations. 
Sixteen  two-person  aircrews  were  battle-rostered  for  the  testbed. 

The  first  step  in  the  testbed  was  for  project  staff  trainers 
to  instruct  the  participating  IPs  and  UTs.  After  receiving 
instruction,  the  IPs  (not  UTs)  rated  the  16  battle-rostered 
participating  aircrews  during  a  full  (premission,  flight,  and 
postmission)  simulator  session.  This  was  the  ''pretraining" 
evaluation  designed  as  the  baseline  against  which  performance 
improvements  would  be  measured.  The  16  aircrews  were  divided 
into  two  groups  of  8  for  the  classroom  instruction.  Two  teams 
each  consisting  of  two  IPs  and  two  UTs  were  formed  to  team-teach 
the  classroom  instruction.  The  instructor  teams  also  instructed 
the  aircrews  during  the  two  course-related  simulator  training 
missions . 

Subsequent  to  the  training,  another  evaluation  mission,  the 
post-training  evaluation  was  given  to  each  of  the  16  aircrews. 

The  eight  crews  given  Scenario  1  for  the  pretraining  mission  were 
given  Scenario  2  for  the  post-training  mission  and  vice  versa. 
Again,  crews  were  rated  using  the  measures  described  above.  When 
the  evaluation  missions  were  completed,  all  testbed  participants 
were  debriefed  on  the  testbed  and  asked  to  critique  the  training. 

Representatives  from  the  crew  coordination  Working  Group  and 
the  project  staff  recorded  observations  and  candidate  lessons 
learned  throughout  the  validation  testbed.  Testbed  participant 
crewmembers  and  trainer-evaluators  contributed  insights  and 
suggestions  during  the  testbed  and  in  their  exit  interviews.  All 
of  the  observations  and  insights  surfaced  during  the  testbed  were 
closely  examined  and  verified  by  additional  sources  and/or 
analysis  of  crew  performance  results  before  being  considered  a 
lesson  learned. 

This  section  provides  a  summary  of  the  lessons  learned 
during  the  validation  testbed  experience  (see  Table  10) .  The 
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Working  Group  (WG) ,  Project  Staff  (PS),  Participants  (P) 


lessons  learned  are  organized  by  the  evaluation  topics  introduced 
in  the  previous  section  to  describe  how  the  evaluation  products 
were  developed.  The  table  includes  a  short  justification  or 
reference  to  the  rationale  for  each  lesson  learned.  Discussion 
of  the  most  important  lessons  learned  and  their  impact  on  the 
Field  Exportable  Evaluation  Package  for  continuation  training 
amplifies  the  summary  table. 

Working  Group  and  Project  Staff 

Working  Group  representatives  attended  the  evaluator 
training,  reviewed  testbed  emerging  results,  and  participated  in 
the  exit  interviews  to  acquire  first-hand  knowledge  about  the 
evaluation  method’s  strengths  and  weaknesses.  Project  staff 
presented  the  evaluator  training  and  administered  the  testbed 
activities . 

Evaluation  lessons  learned  identified  by  the  Working  Group 
and  project  staff  tend  to  be  crew  coordination  program-wide  in 
scope.  For  example,  the  Working  Group  was  particularly 
interested  in  whether  or  not  unit  evaluators  could  comprehend  and 
apply  the  evaluation  method  with  its  thirteen  Basic  Qualities  and 
behaviorially  anchored  rating  scale.  Instructor  pilot-evaluators 
and  unit  trainers  expressed  their  comfort  in  using  the  evaluation 
method  and  confidence  in  the  objectiveness  and  reliability  of 
their  evaluations.  Project  staff  debriefed  testbed  evaluators 
after  every  mission  to  provide  quality  control  of  evaluation 
results  and  collect  comments  on  the  evaluation  method  itself. 
Analysis  of  the  data  collected  using  the  evaluation  method 
demonstrated  high  levels  of  reliability  (see  Testbed  Validation 
Report) . 

Testbed  evaluators  echoed  other  study  findings  regarding  the 
training  and  evaluation  benefit  of  videotaping  crew  performance. 
Review  of  videotaped  missions  during  mision  debriefings  allowed 
evaluators  to  accurately  critique  crew  coordination  actions  and 
resolve  inconsistencies  with  crewmembers.  Working  Group,  project 
staff,  and  testbed  participants  concluded  that  videotaping  and 
playback  of  crew  performance  is  essential  to  train  and  evaluate 
crew  coordination  initial  and  continuation  training. 

The  introduction  phase  of  the  crew  coordination  training 
course  informed  crews  that  the  post-training  evaluation  mission 
would  be  the  course  completion  criteria.  Working  Group  and 
project  staff  recognized  the  need  to  more  clearly  specify  the 
evaluation  requirements  for  course  completion  and  provide 
evaluators  with  candidate  oral  examination  questions. 

Another  area  of  interest  expressed  by  the  Working  Group  was 
the  ability  of  the  13  Basic  Qualities  to  address  all  aspects  of 
crew  coordination.  Working  Group  and  project  staff  assessment  of 
participant  feedback  concluded  that  the  Basic  Qualities 
adequately  describe  all  crew  coordination  actions  and  that  all  of 
the  Basic  Qualities  would  be  demonstrated  in  well  developed 
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scenarios.  It  was  learned  that  including  two  techniques  for 
rating  crew  performance  initially  confused  the  evaluators. 
Analysis  of  evaluator  experience  based  on  having  used  both  the 
rating  factor  and  behavioral  anchor  techniques  concluded  that: 

1.  the  rating  factors  initially  helped  confirm  specific 
Basic  Qualities  for  evaluation  until  the  evaluators  gained 
experience  using  the  behavioral  anchors, 

2.  the  behavioral  anchors  produced  more  reliable 
evaluation  results,  and 

3.  behavioral  anchors  are  preferred  as  the  primary 
criteria  for  evaluating  crew  performance. 

The  Working  Group  examined  the  role  of  ATM  tasks  and  ATM- 
based  grade  slips  in  the  evaluation  method  to  ensure  consistency 
with  implementing  the  revised  ATMs.  Working  Group  and  project 
staff  concluded  that  the  emphasis  on  task  selection  in  the 
scenario  development  guidance  is  key  to  linking  crew  coordination 
evaluations  with  the  ATMs.  This  same  emphasis  must  apply  to 
selection  of  commander  designated  crew  tasks  for  crew 
coordination  continuation  training.  It  is  recognized  that  an  ATM 
task  for  after  action  review  is  needed  for  all  aircraft.  It  was 
further  concluded  that  the  ATM  grade  slips,  revised  to  evaluate 
crew  coordination,  are  understandable  and  easily  adaptable  to 
incorporate  rating  crew  coordination  Basic  Qualities. 

Testbed  Participants 

Instructor-evaluator  and  crewmember  testbed  participants 
provided  project  staff  and  Working  Group  representatives  with 
insights  and  observations  throughout  the  testbed.  Participant 
feedback  tends  to  be  oriented  toward  the  details  of  employing  the 
evaluation  methods  and  products.  The  evaluation  lessons  learned 
shown  in  Table  10  and  discussed  here  are  based  on  analysis  of 
participant  exit  interview  comments  (see  Testbed  Validation 
Report,  Appendix  D) . 

A  primary  lesson  learned  from  participants  is  that  the 
evaluation  method  and  products  are  consistent  with  the  Army's 
current  evaluation  practices.  Crewmembers  and  evaluators  viewed 
the  evaluations  as  professional  and  objective  assessments  of  crew 
rather  than  individual  aviator  performance.  The  rating 
guidelines  and  rating  scale  introduced  to  evaluate  crew 
performance  are  appropriate  for  initial  and  continuation  training 
in  simulators  and  aircraft. 

Evaluators  experimented  using  both  the  rating  factors  and 
behavioral  anchors  techniques  to  evaluate  crew  performance  of 
Basic  Qualities.  Analysis  of  their  experience  concluded  that  the 
rating  factors  are  helpful  at  first  as  abbreviated  descriptions 
of  the  behavioral  anchors  and  that  all  of  the  instructor- 
evaluators  used  the  behavioral  anchors  as  their  primary  rating 
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criteria.  Evaluators  identified  the  need  to  emphasize  the  use  of 
procedure  checklists  in  the  Basic  Quality  descriptions. 

The  modified  grading  system  and  grade  slips  were  effectively 
used  by  unit  instructor-evaluators  during  the  testbed.  The 
expanded  grading  system  using  S+,  S,  S-,  and  U  for  evaluating 
crew  performance  was  well  received  by  evaluators  and  crewmembers. 
Analysis  of  participant  experience  concluded  that  the  expanded 
grading  system  is  an  effective  technique  to  illustrate 
performance  progression,  recognize  exceptional  performance,  and 
better  distinguish  between  marginal  and  unsatisfacto:^ 
performance.  The  expanded  grading  system  is  appropriate  for 
evaluating  crew  performance  in  crew  coordination  initial  and 
continuation  training. 

It  was  also  learned  that  the  simulator  or  aircraft  must  be 
properly  configured  to  react  as  expected  to  the  emergency 
procedures  and  other  tasks  selected  for  evaluation.  Simulator 
set  up  instructions  and  a  scripted  scenario  are  needed  for 
simulator  operators  to  realistically  represent  the  tactical 
situation  and  other  participants  in  the  mission,  for  example, 
nonrated  crewmembers  and  other  aircraft. 

Project  staff  debriefings  with  evaluators  after  each  mission 
disclosed  that  the  overall  mission  grade  is  a  synopsis  of  ATM 
task  grade  slip  entries  and  mission  performance  information 
recorded  on  evaluator  worksheets.  Unit  instructor-evaluators 
used  the  evaluator  workbooks,  provided  as  an  interim  evaluation 
package,  as  a  job  aid  throughout  the  testbed.  Analysis  of  their 
experience  concluded  that  the  evaluator  workbook  is  applicable  to 
all  phases  of  a  mission  and  that  the  evaluator  worksheet  format 
and  organization  is  appropriate  for  continuation  training.  An 
abbreviated  evaluator  worksheet  that  is  reducible  to  grade  slip 
size  is  desirable  for  use  in  the  actual  aircraft.  Evaluators 
identified  the  need  to  add  after  action  review  (AAR)  as  a 
scenario  and  evaluator  worksheet  mission  segment. 

These  lessons  learned  from  using  the  validation  testbed 
evaluation  package  were  incorporated  into  the  Field  Exportable 
Evaluation  Package  design  and  recommendations  for  use  in 
evaluating  crew  coordination  continuation  training. 


Description  of  the  Field  Exportable  Evaluation  Package 

Evaluation  lessons  learned  from  the  validation  testbed  were 
used  to  improve  the  evaluation  process  for  crew  coordination 
initial  and  refresher  training  and  to  develop  an  exportable 
evaluation  process  for  crew  coordination  continuation  training. 
Improvements  to  the  evaluation  process  for  initial  and  refresher 
training  and  the  training  presented  to  evaluators  on  how  to 
conduct  evaluations  are  reported  in  Volume  II,  Development  of 
Candidate  Crew  Coordination  Training  Methods  and  Materials.  This 
section  describes  how  the  testbed  experience  was  used  to  develop 
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the  Field  Exportable  Evaluation  Package,  published  separately, 
and  summarizes  its  contents. 

Development 

The  evaluator's  workbook  developed  for  the  validation 
testbed  served  as  the  interim  or  "trial"  evaluation  package  and 
was  the  basis  for  developing  the  Field  Exportable  Evaluation 
Package  for  continuation  training  (see  Figure  6) . 

Validation  testbed  lessons  learned  endorsed  the  evaluation 
method,  crew  coordination  Basic  Qualities,  and  behaviorally 
anchored  ratings  as  consistent  with  the  evaluation  guidelines 
published  in  the  ATMs.  Improvements  to  the  basic  method  for 
evaluating  crew  performance  emphasized: 

1.  the  nature  of  the  training  being  evaluated,  that  is, 
initial  and/or  refresher  training  or  continuation  training,  and 

2.  the  "crawl -walk- run"  level  of  proficiency  training 
philosophy . 

The  evaluation  method  for  initial  and/or  refresher  training 
concentrated  on  the  evaluator's  instructional  activities  during 
"crawl  and  walk"  training  evaluations  (see  Training  Methods  and 
Materials  Report) .  The  evaluation  method  for  crew  coordination 
continuation  training,  described  below,  is  precisely  aligned  with 
the  sequence  of  activities  for  crewmember  and  crew  flight 
evaluations  published  in  Chapter  8  of  the  ATMs.  This  form  of  the 
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Figure  6.  Development  of  Field  Exportable  Evaluation  Package 
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evaluation  method  stresses  the  "run"  level  of  prof iciencyto  be 
maintained  by  crews  during  continuation  training.  The  basic 
evaluation  method  includes  the  use  of  video  recording  and  oral 
examination  questions  to  achieve  objective,  reliable  evaluations. 

Revised  grade  slips  and  evaluator  worksheet  lessons  learned 
complemented  the  ATM  requirement  that  the  Battle-Rostered  Crew 
Evaluation/Training  Grade  Slip  be  used  to  record  the  results  of 
continuation  training  evaluations.  The  Aircrew  Coordination 
Evaluation  (ACE)  Checklist  was  developed  to  replace  the  Aircrew 
Coordination  Training  Grade  Slip  reserved  for  initial  and/or 
refresher  training  only.  The  ACE  Checklist  provides  for  a 
summa^  rating  of  the  13  Basic  Qualities  using  the  same 
behavioral  anchors  and  rating  scale  as  used  in  initial  training. 
The  evaluator  worksheet  was  modified  to  reflect  commander 
designated  crew  tasks.  The  expanded  grading  system  and 
contributing  Basic  Quality  entries  were  retained  to  provide  a 
means  for  evaluating  continuation  training  with  the  same 
precision  that  was  used  to  certify  completion  of  initial 
training.  Evaluators  are  instructed  to  use  Ace  Checklist  and 
evaluator  worksheet  entries  as  input  to  complete  the  Battle- 
Rostered  Crew  Evaluation/Training  Grade  Slip. 

Contents 

The  Field  Exportable  Evaluation  Package  contains  a  separate 
section  that  addresses  each  of  the  evaluation  topics  abstracted 
below. 

Purpose.  The  express  purpose  of  the  Field  Exportable 
Evaluation  Package  is  to  distribute  evaluation  methods  and 
materials  approved  by  the  USAAVNC  for  units  to  evaluate  crew 
coordination  continuation  training.  Developed  and  tested  in 
conjunction  with  the  Crew  Coordination  Exportable  Training 
Course,  these  methods  and  materials  provide  the  detail  needed  to 
implement  the  evaluation  guidance  published  in  the  Aircrew 
Training  Manuals  (ATM) .  The  package  also  includes  suggestions  on 
applying  crew  coordination  evaluation  results  to  unit  operations. 

Evaluation  Method.  The  method  for  evaluating  crew 
coordination  is  consistent  with  evaluation  guidelines  in  the 
Army's  ATMs.  The  materials  in  the  evaluation  package  provido 
precisely  defined  measures  and  grading  scales  for  the  crew 
coordination  skills  in  the  ATM  standards.  The  crew  coordination 
evaluation  method  complements  the  sequence  of  activities  for 
crewmember  and  crew  flight  evaluations  described  in  Chapter  8  of 
the  ATMs . 

1.  Phase  1  -  Introduction.  In  this  phase,  the  evaluator 
confirms  the  purpose  of  the  evaluation  (for  example,  to  certify 
the  crew's  completion  of  refresher  training,  or  to  demonstrate 
proficiency  in  crew  tasks)  and  discusses  the  criteria  to  be  used 
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(for  example,  ATM  task  standards,  and  Crew  Coordination  Basic 
Qualities) . 


2.  Phase  2  -  Oral  Examination.  Crews  must  have  a  working 
knowledge  and  understanding  of  the  crew  coordination  subject 
area.  The  evaluator  selects  items  from  the  list  of  crew 
coordination  sample  questions  and/or  locally  constructed 
questions . 

3.  Phase  3  -  Flight  Evaluation.  This  phase  consists  of  a 
mission  briefing  and  premission  planning  and  rehearsal,  mission 
execution  in  a  flight  simulator  or  aircraft,  and  a  crew-level 
after-action  review.  Evaluators  use  scenario  materials  to  brief 
the  mission  and  to  provide  the  information  necessary  for  flight 
planning.  While  video  recording  the  crew's  premission  planning 
and  rehearsal,  evaluators  observe  and  rate  performance  of  Crew 
Coordination  Basic  Qualities.  During  mission  execution, 
evaluators  coordinate  scenario  events  and  use  evaluation 
materials,  for  example,  rating  guidance  and  scenario-specific 
evaluator  worksheets  to  observe  and  rate  the  crew's  performance 
of  Crew  Coordination  Basic  Qualities. 

4.  Phase  4  -  Debriefing.  During  this  phase,  the  evaluator 
video  records  the  crew-level  after-action  review.  The  evaluator 
observes  and  rates  the  crew  before  posting  ratings  and  grades  to 
the  grade  slip.  Evaluators  then  use  their  worksheets  and  play 
back  selected  portions  of  the  videotapes  during  the  crew 
debriefing. 

Instructions  for  Use.  This  section  provides  users  with 
instructions  on  how  to  apply  the  methods  and  materials  in  the 
package.  Specific  instructions  are  included  in  the  discussion  of 
each  crew  coordination  evaluation  item  in  the  package.  If 
differences  exist  between  the  evaluation  guidance  in  TC  1-210 
and/or  the  ATMs  and  the  evaluation  package,  TC  1-210  and  the  ATMs 
take  precedence.  For  example,  the  following  instructions  apply 
to  the  evaluation  techniques  and  tools  included  in  the  package: 

1.  Grade  Slips.  Supplement  the  Battle-Rostered  Crew 
Evaluation/Training  Grade  Slip  (DA  7121-R)  with  the  Aircrew 
Coordination  Evaluation  (ACE)  Checklist  to  record  Crew 
Coordination  Basic  Quality  ratings. 

2.  Evaluator  Worksheet.  Develop  scenario-specific 
worksheets  to  sequence  crew  tasks  and  record  notes  for  grade  slip 
preparation . 

3.  Mission  Performance  Measures.  Select  appropriate 
measures  that  relate  crew  coordination  evaluation  results  to  unit 
operations,  for  example,  rounds  or  missiles  on  target  and 
difference  between  planned  and  actual  time  of  arrival. 


46 


video  Recording  Guidance.  The  evaluation  method 
incorporates  video  and/or  audio  recording  and  playback  of 
crewmember  interactions.  Video  and/or  audio  playback  and  review 
of  aircrew  actions  has  proven  to  be  a  powerful  training  and 
evaluation  technique  for  crew  coordination,  making  it  possible  to 
record  the  aircrew  during  all  phases  of  a  mission  (that  is, 
premission  planning  and  rehearsal,  mission  execution  in  the 
simulator  or  aircraft,  and  crew-level  after-action  review) .  The 
types  of  audio-visual  equipment  available  to  aircrew  coordination 
continuation  training  evaluators  will  determine  the  extent  to 
which  this  evaluation  opportunity  is  realized.  The  evaluation 
package  provides  a  table  of  representative  equipment  needed  and 
potential  sources. 

Scenario  Development  Guidance.  This  section  of  the  package 
provides  general  and  specific  guidance  for  developing  realistic 
tactical  scenarios  to  accurately  evaluate  crew  coordination  in 
simulators  and  aircraft.  The  information,  in  this  section 
interprets  and  extends  the  broad  guidance  on  developing  scenarios 
contained  in  doctrine  and  training  literature.  Guidelines  for 
scenario  developers  include  detailed  development  steps, 
information  sources,  and  sample  scenario  products. 

Evaluation  Techniques  and  Tools .  The  package  provides  a 
section  of  detailed  information  on  the  techniques  and  tools 
approved  for  evaluating  crew  coordination  in  continuation 
training.  The  techniques  (evaluation  process,  mission 
performance  measures,  etc.)  and  tools  (ACE  Checklist,  Evaluator 
Worksheet,  etc.)  are  designed  to  accommodate  all  Army  aircraft-- 
rotary  and  fixed  wing.  This  section  includes  descriptions  and 
instructions  for  using  grade  slips,  the  ACE  Checklist, 
effectiveness  factors,  rating  scale  and  rating  guidelines,  Basic 
Qualities  and  behavioral  anchors,  and  evaluator  worksheets. 

Unit  Operations.  This  section  of  the  package  provides 
information  and  suggests  ways  to  apply  crew  coordination 
evaluation  results  to  unit  training  and  operations.  Techniques 
and  tools  are  offered  to  communicate  insights  gained  during  the 
research  and  development  phase  of  the  Crew  Coordination  Program 
that  can  benefit  aviation  unit  operations. 


Recoinmendations  for  Exportable  Evaluation  Package  fielding.  Use, 

and  Improvement 

This  section  provides  a  summary  of  recommendations  for 
fielding,  applying,  and  improving  the  Field  Exportable  Evaluation 
Package.  It  recommends  actions  to  authorize  and  require  units  to 
use  the  evaluation  package  and  describes  ways  to  apply  crew 
coordination  evaluation  results  to  unit  training  and  operations. 
The  section  concludes  by  suggesting  actions  that  the  USAAVNC 
and/or  field  units  could  take  to  improve  continuation  training 
evaluations  and  the  crew  coordination  training  program. 
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Fielding  Recommendations 


The  new  ATMs  do  not  provide  sufficient  detail  to  implement 
the  published  evaluation  guidance.  Unit  instructor  pilots  and 
unit  trainers  participating  in  the  validation  testbed  stated  that 
they  understood  the  evaluation  methods  and  were  confident  that 
the  ratings  they  made  were  objective  and  reliable.  Recommend 
that : 


1.  The  Field  Exportable  Evaluation  Package  methods  and 
materials  be  approved  for  use  by  units  to  conduct  crew 
coordination  continuation  training. 

2.  The  Field  Exportable  Evaluation  Package  be  distributed 
to  units  with  instructor  pilots  certified  as  having  completed  the 
Army  Crew  Coordination  Training  Program.  Suggested  minimum 
contents  for  the  exportable  evaluation  package  include: 

♦  Background  on  method  development . 

•  Video  recording  guidance. 

•  Scenario  development  guidance. 

♦  Evaluation  tools  and  techniques,  that  is;  evaluation 

process,  grade  slips,  the  ACE  Checklist,  evaluator  worksheets, 
and  suggested  mission  performance  measures. 

3.  At  least  one  flight  simulator  at  each  facility  be 
equipped  with  a  video  recording  capability  to  support  evaluation 
of  crew  coordination  performance.  Consideration  should  be  given 
to  equipping  designated  aircraft  that  do  not  have  a  visual 
simulator  with  a  similar  video  recording  capability. 

Other  recommendations  for  fielding  the  Exportable  Evaluation 
Package  and  recommendations  for  additional  research  to  test 
evaluation  methods  and  materials  in  conditions  not  analyzed  to 
date  are  presented  in  the  Testbed  Report . 

Applications 

The  Field  Exportable  Evaluation  Package  describes  a  method 
and  procedures  for  units  to  apply  crew  coordination  evaluation 
results  to  assess  crew  risk. 

Risk  Management.  Risk  management  is  a  fully  integrated  part 
of  mission  planning  and  execution  for  peacetime  situational 
training  exercises  and  actual  combat  missions.  Commanders, 
staff,  troop  leaders,  and  individual  soldiers  are  responsible  for 
the  effective  management  of  risk. 

As  commanders  and  staff  apply  the  risk  management  guidance 
in  TC  1-210,  they  should  consider  including  crew  coordination 
evaluation  results  in  their  SOPs  and  programs  to  manage  risk. 
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Crew  risk  assessment  is  central  to  Army  aviation's  three-tier 
approach  to  risk  management:  individual,  crew,  and 
collective/unit  training.  Current  guidance  includes  two 
considerations  for  assessing  crew  risk:  1)  whether  the  crew  is 
battle-rostered  and  2)  how  long  since  the  crew  has  flown 
together. 

Crew  coordination  evaluation  results  provide  a  rich  source 
of  reliable,  objective  information  on  crew  strengths  and 
weaknesses.  Detailed  information  contained  in  the  crew 
coordination  evaluation  grade  slips  and  crew  performance  measures 
is  relevant  and  can  be  included  in  assessing  crew  risk.  The 
evaluation  package  provides  examples  of  the  types  of  crew 
evaluation  results  information  that  could  be  considered  in  risk 
matrices  and  risk  analysis  techniques. 

Suggested  Improvements 

The  Field  Exportable  Evaluation  Package  describes  suggested 
methods  and  procedures  for  improving  continuation  training 
evaluations  by  introducing  mission  performance  measures  and 
strengthening  the  crew  coordination  training  program  by 
incorporating  evaluation  results  into  crew  status  tracking  and 
unit  readiness  reporting. 

Mission  Performance  Measures.  The  validation  testbed 
demonstrated  that  mission  performance  measures  extend  the 
evaluation  of  crew  tasks  to  overall  mission  performance.  ATMs 
emphasize  that _ research  has  shown  direct,  positive  effects  of 
crew  coordination  on  flight  safety  and  mission  performance.  Used 
in  the  USAAVNC  crew  coordination  research  experiments,  mission 
performance  measures  can  help  units  relate  crew  coordination 
evaluation  results  to  unit  operations  (for  example,  rounds  or 
missiles  on  target,  difference  between  planned  and  actual  time  of 
arrival) . 

The  process  for  developing  crew  tasks  described  in  TC  1-210, 
"Commander's  Guide  to  Individual  and  Crew  Training,"  includes  a 
review  of  unit  and  collective  training  publications,  for  example, 
ARTEP  Mission  Training  Plan  (MTP)  and  Battle  Drills.  Analysis  of 
the  unit's  MTP  missions  is  a  guide  to  the  identification  of 
mission  performance  areas  and  specific  mission  performance 
measures  for  evaluation.  Study  of  potential  mission  performance 
areas  is  necessairy  to  ensure  that  they  can  be  realistically 
included  in  scenario  segments  and  evaluated  in  a  flight  simulator 
or  aircraft. 

The  evaluation  package  provides  suggestions  for  developing 
mission  performance  measures,  to  include  example  measurement 
areas  and  measures . 

Crew  Tracking  and  Unit  Readiness.  The  evaluation  package 
suggests  that  units  consider  using  continuation  training 
evaluation  results  to  assist  in  determining  the  status  of  unit 
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crew  coordination  training  and  unit  readiness.  For  example,  TC 

1- 210  requires  that  unit  commanders  maintain  crew  task  iteration 
tracking  sheets  and  crew  grade  slips.  A  Crew  Training  Record  (DA 
Form  7122-R)  is  provided  to  monitor  a  crew's  progress  in 
completing  the  commander's  designated  tasks  and  required 
iterations,  for  example,  day,  night,  and  night  vision  device. 

The  ACE  Checklist  attached  to  each  crew  evaluation  grade 
slip  provides  an  additional  source  of  information  to  track  crew 
progress.  Units  could  record  and  compare  the  rating  column 
entries  across  a  series  of  ACE  Checklists  to  identify  the  crew's 
strengths,  weaknesses,  and  trends  with  respect  to  Basic 
Qualities.  This  information  could  be  used  to  tailor  continuation 
training  for  a  crew  and  provide  focus  areas  for  evaluators. 

The  status  of  crew  proficiency  could  be  determined  by 
totaling  the  ratings  for  each  Basic  Quality  across  all  crews  and 
computing  an  average  for  each  Basic  Quality,  that  is,  sum  the 
rating  values  for  Basic  Quality  1  across  all  crews  and  divide  by 
the  number  of  crews,  then  repeat  the  process  for  Basic  Qualities 

2- 13.  This  provides  a  unit-level  average  rating,  that  is,  (1) 
Very  Poor  to  (7)  Superior  for  each  Crew  Coordination  Basic 
Quality.  This  information  could  be  used  to  identify  a  unit's 
crew  coordination  strengths  and  weaknesses  and  to  assist  in 
determining  the  number  of  days  needed  to  fully  train  to  standard 
on  unit  METL  tasks. 

Crew  coordination  evaluation  results  could  provide 
supporting  data  for  the  commander's  assessment  of  unit  readiness. 
Total  all  Basic  Quality  ratings  for  all  crews  and  compute  a  unit 
average  rating,  that  is,  sum  all  rating  values  for  all  Basic 
Qualities  for  all  crews,  divide  by  13,  and  then  divide  by  the 
number  of  crews.  This  provides  a  unit  average  crew  coordination 
rating,  that  is,  (1)  Very  Poor  to  (7)  Superior.  This  information 
could  be  used  to  support  the  commander's  aviator  training 
readiness  C-rating  based  on  percent  of  RL-1  crews. 
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Appendix  A 

Army  Aviation  Crewmember  Questionnaire 


Instructions 


The  US  Army  Aviation  Center  (USAAVNQ  and  the  US  Army  Research  Institute  (ARI)  are  researching 
the  area  of  crew  coordination  in  Army  Aviation.  The  goal  of  this  research  is  to  improve  performance 
and  increase  the  margin  of  safety  on  an  Army-wide  basis.  Previous  research  by  other  DoD  services  and 
commercial  aviation  into  the  area  of  crew  coordination  has  contributed  to  substantial  gains  in  both 
performance  and  safety. 

Because  Army  Aviation  is  unique,  much  of  the  information  discovered  by  the  other  services  and  the 
commercial  world  is  not  directly  applicable  to  the  Army  Aviation  environment.  Consequently,  the 
USAAVNC-ARI  research  program  is  designed  to  meet  the  specific  needs  of  Army  aviation.  As  a  result 
of  this  approach,  the  following  actions  are  now  ongoing  or  planned:  Mission  simulations  are  being 
developed  to  stress  aircrew-t3q>e  tasks,  enhanced  aircrew  coordination  training  is  being  developed,  the 
US  Army  Safety  Center  is  incorporating  crew  factors  into  the  accident  investigation  process.  Aircrew 
Training  Manuals  and  the  aimual  proficiency  and  readiness  test  program  are  being  revised,  and  revisions 
to  readiness  reporting  are  being  planned. 

This  Army  Aviation  Crewmember  Questionnaire  has  been  developed  as  part  of  the  USAAVNC-ARI 
research  program  to  obtain  your  opinion  about  crew  operations.  As  an  Army  aviator,  your  participation 
is  essential  to  the  program’s  success.  Your  opinions  are  important  and  will  be  used  to  guide  the  next 
phase  of  the  research  program. 

The  questioimaire  should  take  approximately  20  minutes  to  complete.  The  first  page  of  the  questionnaire 
asks  you  for  background  information  ~  please  try  to  be  accurate.  The  next  three  pages  contain  46 
statements  for  which  there  are  no  "right"  or  "wrong"  answers.  We  are  simply  asking  for  your  honest 
opinion  to  each  statement.  Please  consider  each  statement  carefully. 

THANK  YOU  FOR  YOUR  PARTICIPATION. 


IMPORTANT 

The  information  you  provide  in  this  questionnaire  is 
confidential  and  will  be  used  for  research  purposes 
only.  Your  answers  will  neither  be  attributed  to  you 
personally  nor  become  a  part  of  any  personnel  or 
aviation  record  kept  on  you. 
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Army  Ayiation  Crewmember  Questionnaire 


I.  Background  Information 

(Please  complete  the  following  information  regarding  your  personal  experiences  and  current  status.) 

1 .  Aviation  Experience: 

Lifetime  Flving  Experience  Experience  over  last  6  months 

All  NV  Devices  All  NV  Devices 

Conditions  (e.g.,  NVG)  Conditions  (e.g.,  NVG) 

a.  Primary  acft  hrs.  _  _  _  _ 

b.  R/W  hrs.  _  _  _  _ 

c.  Fixed  Wing  hrs.  _  _  _  _ 

2.  Primary  Aircraft _ (Fill  in  aircraft  designation) 

3.  Current  Rank _ 

4.  Current  Unit  (Co/Bn/Rgt) _ 

5.  Time  in  Current  Unit  (months) _ 

6.  Current  Aviator  Readiness  Level  (RL)  1  2  3  (circle  one  number) 

7.  Current  Crew  Readiness  Level  (CRL)  1  2  (circle  one  number) 

8.  Current  primary  duty  assignment  in  unit  (check  one)'. 

PC* _  PI _  CP _  CPG _  CE/FE _  AO/AFSO/TO _  OR* _ 

*Note:  PC  includes  IP,  SP,  IE,  UT,  ME,  MP  duty  positions;  OR  includes  gunner  and  flight  medic. 

9.  Are  you  flight  lead  qualified  (circle  one):  Yes  No 

10.  Have  you  had  Aircrew  Coordination  Training?  Y  or  N  (circle  one:  if  yes,  answer  below.) 

Describe  ACT  training  experiences:  Course  title,  location  of  training,  approximate  date,  ft  of  hours  of 
instruction,  quality  of  course. 

a.  Experience  ft\ :  _ 


b.  Experience  #2: 


1 1 .  Cross-indexing  Code  (Note:  Because  the  results  of  this  questionnaire  will  be  correlated  with  other 
measures,  a  social  security  number  is  required.) 

Social  Security  #:  _  Today’s  Date  _ 

(day/mo/yr) 


A-2 


Rev.  5 


n.  Opinion  Survey 

(Please  circle  the  number  on  the  agree-disagree  dimension  that  best  reflects  vour  personal  attitude 
toward  each  statement.  There  are  no  "right"  or  "wrong "  answers.  We  are  simply  asking  for  your 
honest  opinions.) 

Strongly  Disagree  Slightly  Neutral  Slightly  Agree  Strongly 
Disagree  Disagree  Agree  Agree 

1.  Crewmembers  should  feel  obligated  to  mention  their  own  1  2  3  4  5  6  7 

psychological  stress  or  physical  problems  to  other  crewmembers 

before  or  during  a  mission. 

2.  Crewmembers  should  monitor  each  other  for  signs  of  stress  orl  2  34567 

fatigue  and  should  discuss  the  situation  with  the  affected 

crewmember(s)* 

3.  Good  communication  and  crew  coordination  are  as  important  1  2  3  4  5  6  7 

as  technical  proHciency  for  the  safety  of  the  flight. 

4.  Crewmembers  should  be  aware  of  and  sensitive  to  the  personal  1  2  3  4  5  6  7 

problems  of  other  crewmembers. 

5*  The  pilot  flying  the  aircraft  should  verbalize  plans  for  1  2  3  4  5  6  7 

procedures  or  maneuvers  and  should  be  sure  that  the 
information  is  understood  and  acknowledged  by  affected 
crewmembers. 

6.  Even  when  fatigued,  I  perform  elTectively  durii^  most  critical  1  2  3  4  5  6  7 

flight  maneuvers. 

7.  Pilots-in-command  should  encourage  pUots  and  crew  chiefs  tol  2  34567 

question  procedures  and  flight  profile  deviations  during  normal 

flight  operations  and  in  emergencies. 

8.  There  are  no  circumstances  where  the  pilot  should  take  the  1  2  34567 

aircraft  controls  without  being  directed  to  do  so  by  the  pilot-in¬ 
command. 

9.  A  debriefing  and  after  action  review  of  procedures  and  1  2  3  4  5  6  7 

decisions  after  each  mission  are  important  for  developing  and 

maintaining  effective  crew  coordination. 

10.  Crew  coordination  is  more  important  under  high  stress  1  2  3  4  5  6  7 

conditions  than  it  is  under  low  stress  conditions. 

11.  Effective  crew  coordination  requires  crewmembers  to  take  into  1  2  3  4  5  6  7 

account  the  personalities  of  other  crewmembers. 
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Strongly  Disagree  Slightly  Neutral  Slightly  Agree  Strongly 
Disagree  Disagree  Agree  Agree 


12.  The  pilot-in-cominand’s  responsibilities  include  coordinating  1  2  3  4  5  6  7 

inflight  crew  chief  activities. 

13.  Most  crewmembers  are  able  to  leave  personal  problems  behind  1  2  3  4  5  6  7 

when  flying  a  mission. 

14.  My  decision  making  ability  is  as  good  in  emergencies  as  it  is  in  1  2  3  4  5  6  7 

routine  mission  situations. 

15.  The  pilot-in-command  is  solely  responsible  for  leadership  of  the  1  2  3  4  567 

crew  team. 

16.  Pilots  should  consider  crew  chief  questions  and  suggestions.  1  2  3  4  5  6  7 

17.  When  joining  a  unit,  a  new  crewmember  should  not  offer  1  2  3  4  5  6  7 

suggestions  or  opinions  unless  asked. 

18.  Because  crew  chiefs  have  no  pilot  training,  they  should  limit  1  2  3  4  5  6  7 

their  attention  to  their  formally  defined  crew  chief  duties. 

19.  Pilots-in-command  who  accept  and  implement  suggestions  from  1  2  3  4  5  6  7 

the  crew  lessen  their  stature  and  reduce  their  authority. 

20.  Crewmembers  should  monitor  the  pilot-in-command’s  1  2  3  4  5  6  7 

performance  for  possible  mistakes  and  errors. 

21.  The  best  way  to  correct  an  error  is  to  alert  the  error  maker  sol  2  34567 

that  he  can  correct  the  problem. 

22.  Crewmembers’  errors  and  mistakes  during  the  mission,  1  2  3  4  5  6  7 

including  the  pilot-in-command’s  mistakes,  should  be  a 

significant  part  of  post  flight  crew  discussions. 

23.  The  pilot-in-€ommand  should  seek  advice  from  crewmembers  1  2  3  4  5  6  7 

when  updating  mission  plans. 

24.  The  pUot-in-command  should  use  his  crew  to  help  him  maintain  1  2  3  4  5  6  7 

situation  awareness. 

25.  The  pilot-in-command  is  solely  responsible  for  maintaining  1  2  3  4  5  6  7 

awareness  of  crew  capabilities. 

26.  Only  when  the  pilot-in-command  is  overloaded  should  he  pass  1  2  3  4  5  6  7 

workload  to  other  crewmembers. 
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Strongly 

Disagree 

Disagree 

Slightly 

Disagree 

Neutral 

Slightly 

Agree 

Agree 

Strongly 

Agree 

27. 

Crewmembers  should  be  aware  of  other  crewmembers’ 
workload. 

1 

2 

3 

4 

5 

6 

7 

28. 

If  a  crewmember  is  having  dii^culties  executing  his 
responsibilities,  other  crewmembers  should  provide  assistance. 

1 

2 

3 

4 

5 

6 

7 

29. 

EQghly  competent  pilots  do  not  experience  task  overload. 

1 

2 

3 

4 

5 

6 

7 

30. 

A  crewmember  should  offer  task  help  to  another  crewmember 
only  if  he  is  sure  the  crewmember  needs  it. 

1 

2 

3 

4 

5 

6 

7 

31. 

The  pUot-in-command  should  not  get  involved  with  the 
execution  of  responsibilities  assigned  to  other  crewmembers. 

1 

2 

3 

4 

5 

6 

7 

32. 

Crewmember  task  overload  usually  occurs  because  the 
crewmember  is  not  very  competent. 

1 

2 

3 

4 

5 

6 

7 

33. 

Pilots-in-command  should  employ  the  same  style  of  leadership 
in  all  situations  and  with  all  crewmembers. 

1 

2 

3 

4 

5 

6 

7 

34. 

PUot-in-command  instructions  to  other  crewmembers  should  be 
general  and  non-specific  so  that  each  individual  can  practice 
self-management  and  can  develop  individual  skills. 

1 

2 

3 

4 

5 

6 

7 

35. 

A  relaxed  attitude  is  essential  for  maintaining  a  cooperative  and 
harmonious  cockpit. 

1 

2 

3 

4 

5 

6 

7 

36. 

Reprimands  are  more  elective  than  discussions  in  eliminating  a 
crewmember’s  poor  flying  habit. 

1 

2 

3 

4 

5 

6 

7 

37. 

Nonrated  crewmembers  should  be  actively  involved  in  planning 
the  mission. 

1 

2 

3 

4 

5 

6 

7 

38. 

Understanding  the  commander’s  concept  is  of  minor 
importance  to  mission  execution. 

1 

2 

3 

4 

5 

6 

7 

39. 

Each  crewmember  should  watch  for  situations  in  which 
external  events  limit  others’  performance. 

1 

2 

3 

4 

5 

6 

7 

40. 

Thinking  through  difficult  segments,  events,  and  tasks  is 
primarily  the  pUot-in-command’$  responsibility. 

1 

2 

3 

4 

5 

6 

7 

41. 

My  knowledge  of  unit  SOP  and  aircraft  emergency  procedures 
makes  rehearsing  familiar  missions  unnecessary. 

1 

2 

3 

4 

5 

6 

7 
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Strongly 

Disagree 

Disagree 

Slightly 

Disagree 

Neutral 

Slightly 

Agree 

Agree 

Strongly 

Agree 

42. 

An  essential  element  of  premission  planning  is  discussing  crew 
responsibilities  and  required  actions  for  abnormal  events. 

1 

2 

3 

4 

5 

6 

7 

43. 

Recent  events  in  my  personal  life  have  little  to  do  with  my 
performance  as  a  crewmember. 

1 

2 

3 

4 

5 

6 

7 

44. 

Crewmembers  should  be  able  to  anticipate  requirements  as  the 
mission  progresses. 

1 

2 

3 

4 

5 

6 

7 

45. 

My  individual  performance  is  as  good  in  degraded  systems 
conditions  as  it  is  in  a  ”fiill  up”  aircraft. 

1 

2 

3 

4 

5 

6 

7 

46. 

External  circumstances  require  crewmembers  to  provide 
situational  leadership  for  short  periods  of  time. 

1 

2 

3 

4 

5 

6 

7 
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Appendix  B 

Aircrew  Coordination  Rating  Guidelines  and  Basic  Qualities 


Note:  The  Aircrew  Coordination  Evaluation  (ACE)  Checklist, 
developed  for  use  in  the  Field  Exportable  Evaluation  Package,  is 
presented  here  as  an  index  to  the  crew  coordination  Basic 
Qualities 


Rating  Scale 


The  following  numeric  rating  scale  is  used  to  assess  the  level  of 
behavior  that  crews  exhibit  for  each  basic  quality  shown  on  the 
Aircrew  Coordination  Evaluation  (ACE)  Checklist  (see  Figure  B-1) 
and  at  the  bottom  of  the  Aircrew  Coordination  Training  Grade 
Slip.  Each  basic  quality  is  rated  using  a  seven-point  scale  with 


values 

rdnging  from  1  (very  poor)  to 

7  (superior) : 

Very 

Poor 

Poor 

Accept - 

Marginal  able 

Good 

Very 

Good 

Superior 

1 

2 

3  4 

5 

6 

7 

Ratina 

Guidelines 

Written  descriptions  of  the  types  of  behaviors  and  levels  of 
performance  are  shown  for  rating  values  1,  4,  and  7.  These 
descriptions  serve  as  behavioral  "anchors"  and  are  designed  to 
assist  evaluators  in  determining  how  well  a  crew  performs  on  each 
basic  quality  in  relation  to  a  well-defined  set  of  behaviors. 
Evaluators  should  use  the  "anchors"  as  the  standard  for  making 
ratings --avoid  comparing  one  crew's  performance  with  that  of 
another  crew's;  rate  a  crew's  performance  in  relation  to  the 
"anchors."  To  ensure  reliable  ratings,  continue  to  refer  to  the 
anchors  when  making  rating  responses  until  completely  confident 
and  understand  fully  how  to  rate  each  basic  quality. 

In  completing  a  basic  quality  rating,  evaluators  should  decide 
whether  the  behaviors  observed  fall  into  the  low  end  of  the  basic 
quality  range  (values  1  or  2),  the  middle  of  the  range  (values  3, 
4,  or  5) ,  or  the  high  end  of  the  range  (values  6  or  7) .  Once  the 
general  range  of  response  is  selected,  use  the  anchors  to  help 
select  the  final  rating  value.  For  example,  if  a  crew  did  an 
adequate  job  of  pre-mission  planning  and  rehearsal,  the  rating 
would  come  from  the  middle  of  the  range  (3,  4,  or  5) .  After 
determining  this,  review  the  behavioral  description  (anchor) 
associated  with  value  4  to  determine  if  crew  performance 
resembled  this  description  (4  value) ,  was  somewhat  less  than  this 
description  (3  value),  or  was  a  little  better  than  this 
description  (5  value) .  Use  the  end-point  anchors  similarly  to 
help  determine  ratings  that  fall  near  the  ends  of  the  scale. 

Ariry  aviation  crews  that  have  little  or  no  training  in  aircrew 
coordination  techniques  will  score  most  frequently  in  the  lower 
half  of  the  scale.  Most  other  crews,  however,  will  fall  into 
the  middle  area  of  the  scale.  Keep  in  mind  that  although  Army 
aviators  have  well  developed  basic  flying  skills,  as  a  group, 
their  aircrew  coordination  skills  will  be  much  like  the 
rest  of  the  population.  A  few  crews  will  have  strong 
coordination  and  communication  skills,  a  few  will  have  weak 
skills,  and  a  significant  number  will  have  moderate  skills. 
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Figure 


AIRCREW  COORDINATION  EVALUATION  (ACE)  CHECKLIST 

For  use  of  this  form,  see  Aircrew  Coordination  Exportable  Evaluation 
Package  for  Army  Aviation. 


PC. 

PI  - 


Date 


NCM  . 


NO 


CREW  COORDINATION  BASIC  QUALITIES 


RATING 


Establish  and  maintain  flight  team  leadership  and  crew  climate 
(Crew  Climate) 


Premission  planning  and  rehearsal  accomplished  (Plan  Rehearse) 


Application  of  appropriate  decision  making  techniques  (Decision  Tech) 


Prioritize  actions  and  distribute  workload  (Workload) 


Management  of  unexpected  events  (Unexp  Events) 


Statements  and  directives  clear,  timely,  relevant,  complete,  and  verified 
(Info  Xfer) 


Maintenance  of  mission  situational  awareness  (Sit  Aware) 


Decisions  and  actions  communicated  and  acknowledged  (Comm/Ack) 


Supporting  information  and  actions  sought  from  crew  (Info  Sought) 


10 


Crewmember  actions  mutually  cross-monitored  (Cross  Monitor) 


11 


Supporting  information  and  actions  offered  by  crew  (info  Offered) 


12 


Advocacy  and  assertion  practiced  (Advoc/Assert) 


13  Crew-level  after-action  reviews  accomplished  (AAR) 


Evaluator's  Signature; 


Notes: 

Consult  the  behavioral  anchored  rating  guidance.  Enter  a  summary  rating  (1 , 2  ...  7)  in  the  rating 
block  for  each  Basic  Quality.  Refer  to  the  rating  scale  below. 


RATING  SCALE 


Very  Poor 

Poor 

Marginal 

Acceptable 

Good 

Very  Good 

Superior 

1 

2 

3 

4 

5 

6 

7 

AIRCREW  COORDINATION  EVALUATION  (ACE)  CHECKLIST 

B-l.  Behavioral  anchored  ratings 
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Aircrew  Coordination  Basic  Qualities  and  Behavioral  Anchors 


BASIC  QUALITY  1.  EststbliBh  and  maintain  flight  teem  leadership 

and  crew  climate  (Crew  Climate) 


Explanation: 

This  rating  assesses  the  quality  of  relationships  among  the  crew 
and  the  overall  climate  of  the  flight  deck.  Aircrews  are  teams 
with  a  designated  leader  and  clear  lines  of  authority  and 
responsibility.  The  pilot -in-command  sets  the  tone  of  the  crew 
and  maintains  the  working  environment.  Effective  leaders  use 
their  authority  but  do  not  operate  without  the  participation  of 
other  crewmembers.  When  crewmembers  disagree  on  a  course  of 
action,  rate  the  crew's  effectiveness  in  resolving  the 
disagreement.  Note:  Traditional  leadership  centralizes 
leadership  in  the  leader  with  followers  fully  dependent  on  the 
leader.  Functional  leadership  assigns  leadership  and 
followership  roles  as  the  situation  evolves.  Flight  team 
leadership  recognizes  the  impact  of  leadership  style  on  the 
working  environment.  Regardless  of  leadership  style,  the  pilot- 
in-command  retains  final  decision  and  direction  authority. 

Superior  Rating  (7) 

The  crewmembers  have  ve:^  good  interpersonal  relationships .  They 
respect  each  others'  skills  and  appear  to  enjoy  being  with  each 
other.  The  climate  is  very  open;  crewmembers  freely  talk  and  ask 
questions.  Crewmembers  encourage  the  individual  with  the  most 
information  about  the  situation-at-hand  to  participate.  There  is 
a  genuine  concern  for  good  working  relationships .  No  degrading 
comments  or  negative  voice  tones  are  used  in  interactions. 
Disagreements  are  perceived  as  a  normal  part  of  crew 
interactions,  and  the  crew  directly  confronts  the  issues  over 
which  the  disagreement  began.  Arguments  or  disagreements  focus 
on  behaviors  or  solutions  rather  than  on  personalities.  Each 
crewmember  carefully  listens  to  others'  comments.  Senior 
crewmembers  accept  challenges  from  junior  crewmembers. 

Alternative  solutions  are  explored.  The  solution  produced  is  a 
"win-win"  situation  in  which  all  crewmembers'  opinions  are 
considered.  The  crewmembers  have  no  hard  feelings  at  the 
conclusion  of  the  incident. 


Acceptable  Rating  (4) 

The  crewmembers  have  sound  interpersonal  relationships  and  seem 
to  respect  each  others'  skills.  The  climate  is  an  open  one,  and 
crewmembers  are  free  to  talk  and  ask  mission  questions. 

Regardless  of  rank  or  duty  position,  the  individual  with  the  most 
information  about  the  situation-at-hand  is  allowed  to 
participate.  When  disagreements  arise,  the  crew  directly 
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confronts  the  issues  over  which  the  disagreements  began.  The 
primary  focus  is  on  behaviors  or  solutions,  and  no  personal 
attacks  are  made  in  the  heat  of  discussion.  The  solution  is 
generally  seen  as  reasonable.  Problem  resolution  ends  on  a 
positive  note  with  very  little  hostility  or  grumbling  among 
crewmembers.  Mutual  respect  is  clearly  intact. 


veiry  Poor  Rating  (1) 

Crew  interactions  are  often  awkward  and  uncomfortable.  The 
crewmembers  do  not  appear  to  like  or  respect  each  other. 
Crewmembers  may  be  curt  and  impolite  to  each  other.  Requirements 
for  assistance  are  made  as  commands  rather  than  as  requests  for 
support.  When  disagreements  arise,  the  crew  fails  to  directly 
confront  the  issues.  Personal  attacks  may  arise.  Senior 
crewmembers  are  resistant  to  recommendations  from  junior 
crewmembers.  Crewmembers  do  not  explore  the  range  of  possible 
solutions.  They  may  shout  and  argue  without  finding  a  solution. 
One  or  more  crewmembers  may  retreat  and  say  nothing  at  all.  A 
"win-lose"  situation  develops  in  which  one  crewmember  is  shown  to 
be  right  and  the  other  to  be  wrong.  The  crewmembers  show  little 
respect  to  one  another  except  for  deferring  to  formal  rank. 


BASIC  QUALITY  2.  Pre-miasion  planning  and  rehearsal 

accomplished  (Plan  Rehearsal) 


Explanation: 

This  rating  assesses  the  pre-mission  planning  and  rehearsal 
activities  that  the  crew  performs  upon  receiving  a  mission  order. 
Time  available  determines  whether  pre-mission  planning  and 
rehearsal  is  completed  prior  to  the  flight  or  in  the  cockpit. 
During  this  period  crews — 

•Clarify  the  mission  order  and  the  commander's  intent 
•Assign  actions,  duties,  and  mission  responsibilities 
•Collect  information  (intelligence,  communications, 
weather,  flight  planning)  and  develop  the  plan 
•Conduct  crew  briefing  to  review  and  discuss  the  plan 
•Identify  potential  problem  areas  and  courses  of  action 
•Assess  risks 

•Visualize  and  rehearse  the  mission 

Although  the  pilot-in-command  is  responsible  for  leading  this 
activity,  evaluate  the  extent  and  manner  in  which  the  entire  crew 
participates.  Also,  consider  the  time  constraints  on  the  crew.  If 
there  is  insufficient  time  to  conduct  comprehensive  planning  and 
rehearsal,  evaluate  the  crew  on  their  planning  and  rehearsal  of 
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the  most  critical _ segments  of  the  mission.  That  is,  either  prior 
to  the  flight  or  in  the  cockpit,  did  the  crew  address  the  most 
important  issues  given  the  time  available?  Note:  The 
relationship  among  crew  members  should  be  observed  during  this 
period  but  the  crew  climate  evaluation  should  be  made  on  rating 
basic  quality  1,  Flight  Team  Leadership  and  Crew  Climate. 

Examp 1 es : 

•UH-60  Task  2078  and  AH-64  Task  1033,  Perform  terrain  flight 
mission  planning:  The  crew  will  analyze  the  mission  in  terms  of 
METT-T  and  plan  the  flight  as  directed  by  the  PC.  The  crew  will 
rehearse  important  aspects  of  the  mission. 

•UH-60  and  AH-64  Task  1000,  Conduct  crew  mission  briefing: 

Aircrew  collectively  visualizes  and  rehearses  expected  and 
unexpected  events  from  takeoff  to  tie-down;  all  factors  of  the 
flight;  and  actions,  duties,  and  responsibilities  of  each 
crewmember. 

•AH-64  and  UH-60  Task  1068,  Perform  or  describe  emergency 
procedures:  PC  will  include  in  the  crew  briefing  the  general 
approach  to  all  emergency  procedures  requiring  immediate  action. 

Superior  Rating  (7) 

The  entire  crew  discusses  a  detailed  description  of  the  mission 
and  the  commander's  intent.  All  actions,  duties,  and  mission 
responsibilities  are  partitioned  and  clearly  assigned  to  specific 
individuals .  The  crew  acquires  new  and  updated  information  and 
uses  it  to  develop  the  mission  plan  from  the  aircrew  mission 
briefing.  Questions  and  discussion  about  the  mission,  commander's 
intent,  and  specific  responsibilities  are  encouraged.  Potential 
problems  are  noted  and  discussed  in  detail.  Courses  of  action  and 
individual  responsibilities  are  established  in  the  event  that 
potential  problems  actually  occur.  All  crewmembers  speak  out  and 
acknowledge  an  understanding  of  the  operational  risks  in  the 
mission  plan.  The  pilot-in-command  leads  the  crew  in  mentally 
rehearsing  the  entire  mission  by  visualizing  and  talking  the  crew 
through  potential  problems  and  contingencies.  Crewmembers 
acknowledge  understanding  their  assigned  responsibilities  and 
cues  for  actions.  The  tone  of  the  interaction  is  friendly  and 
professional . 


Acceptable  Rating  (4) 

A  brief  description  of  the  mission  is  provided  to  the  entire 
crew.  The  mission  responsibilities  are  partitioned  and  assigned 
to  specific  individuals.  Actions  are  taken  to  update  current 
information  that  adds  to  the  aircrew  mission  briefing  and  helps 
develop  the  mission  plan.  One  or  more  crewmembers  make  comments 
during  the  course  of  developing  the  mission  plan.  Potential 
mission  problems  are  only  briefly  discussed.  There  is  adequate 
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preparation  for  contingencies.  Crewmembers  briefly  discuss  the 
operational  risks  in  the  mission  plan.  Mental  rehearsal  is 
initiated  by  the  pilot-in-command  or  another  crewmember  who  talks 
through  potential  problems  or  contingencies  for  one  or  more 
mission  segments.  Some  discussion  takes  place  to  clarify 
responsibilities  in  the  event  of  unexpected  problems  or 
contingencies.  The  tone  of  the  interaction  is  generally  friendly 
and  businesslike. 


Very  Poor  Rating  (1) 

The  pilot -in-command  briefs  the  mission  with  little  or  no 
attendant  explanation.  There  is  little  or  no  discussion  of 
responsibilities  or  their  assignments  to  specific  crewmembers. 

The  pilot-in-command  develops  the  mission  plan  from  the  aircrew 
mission  briefing  and  current  information.  Crewmembers  tend  not  to 
ask  questions  about  the  mission.  If  asked,  questions  tend  to  be 
cut  off,  only  briefly  addressed,  or  ignored  by  the  other  crew¬ 
members.  Little  or  no  mention  is  given  to  potential  problems  or 
complications.  No  crewmember  says  anything  about  operational 
risks  or  weaknesses  in  the  plan.  Any  suggestion  to  talk  through  a 
potential  problem  or  mentally  rehearse  responsibilities  is 
rejected  as  unnecessary.  The  tone  of  the  interaction  is 
business-like,  abrupt,  and  impersonal. 


BASIC  QUALITY  3.  Application  of  appropriate  deal -making 

tecbnlqpies  (Decision  Tech) 


Explanation: 

This  rating  evaluates  the  manner  and  quality  of  the  crew's 
problem  solving  and  decision  making  performance  throughout  the 
planning  and  execution  of  the  mission.  Factors  to  consider  in 
making  this  evaluation  include  (1)  information  available  to  the 
crewmembers,  (2)  time  urgency  of  the  decision,  (3)  objectivity 
reflected  in  the  decision  process,  and  (4)  level  of  involvement 
and  information  exchange  among  the  crewmembers.  The  time 
critical  demands  of  tactical  flying  require  many  decisions  to  be 
made  on  an  automatic,  pattern-recognition  basis  with  only  a 
minimum  level  of  information  exchange.  However,  when  adequate 
time  and  information  are  available,  crewmembers  are  expected  to 
engage  in  a  more  deliberate  and  interactive  style  of  decision 
making.  The  evaluation  of  crew  decision  making  performance 
should  ask  the  following  questions:  (1)  Did  the  crew  use  all  of 
the  available  information?  (2)  Was  the  level  of  information 
exchange  among  crewmembers  appropriate  for  the  time  available? 
(3)  Was  the  type  of  decision  process  (deliberate  versus 
automatic)  appropriate  for  the  time  available? 
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Examples: 

•UH-60  and  AH-64  Task  2044,  Perform  actions  on  contact:  Crew  will 
discuss  options  for  developing  the  situation,  then  choose  a 
course  of  action  that  supports  the  intent  of  the  unit  commander's 
directives. 

•AH-64  and  UH-60  Task  2083,  Negotiate  wire  obstacles:  Crew  will 
discuss  the  characteristics  of  the  wires  ...  to  determine  the 
method  of  crossing. 

Superior  Rating  (7) 

Crew  decision  making  consistently  reflects  proper  attention  to 
available  information  throughout  mission  planning  and  execution. 
The  level  of  crew  participation  and  deliberate  analysis  of 
options  is  appropriate  for  the  decision  time  available.  Resulting 
decisions  are  timely  and  appropriate  given  the  time  urgency  and 
level  of  information  available  in  each  situation.  Crewmembers  do 
not  exhibit  any  of  the  known  hazardous  thought  patterns  (e.g., 
anti-authority,  impulsivity,  machoism,  invulnerability, 
resignation,  get-home-itis,  overconfidence  in  other  aviator)  and 
appear  motivated  to  seek  the  most  mission  effective  and  safe 
decision  in  each  situation.  The  crew  decides  and  implements  a 
course  of  action  before  the  situation  jeopardizes  crew 
performance  or  mission  accomplishment. 


Acceptable  Rating  (4) 

Crew  decisions  occasionally  reflect  inadequate  sharing  or  use  of 
available  information.  On  limited  occasions,  crewmembers  dwell 
excessively  on  some  issues  while  neglecting  more  time  urgent 
requirements.  Most  decisions  are  timely,  but  crew  performance 
begins  to  show  signs  of  self-induced  stress.  Most  decisions  are 
appropriate  for  the  situation,  with  the  crew  occasionally 
overlooking  one  or  more  factors  or  options.  Crewmembers 
occasionally  fail  to  recognize  or  exploit  opportunities  for 
additional  planning  or  rehearsal,  substituting  instead  ad  hoc 
strategies  or  plans .  Crewmembers  do  not  exhibit  any  of  the  known 
hazardous  thought  patterns.  The  situation  may  worsen,  without 
seriously  degrading  mission  accomplishment,  before  the  crew 
decides  and  implements  a  course  of  action. 


Very  Poor  Rating  (1) 

Crew  performance  (both  pre-flight  and  in-flight)  reflects  an 
inflexible  style  of  decision  making  (either  deliberate  or 
automatic)  regardless  of  time  urgency.  Crewmembers  may  engage  in 
excessive  deliberation,  overlook  the  relative  time  urgency  of 
competing  decision  requirements,  or  fall  victim  to  inappropriate 
mind  sets.  As  a  result,  decisions  frequently  lack  timeliness, 
ignore  important  factors,  or  appear  out  of  context.  Information 
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exchange  and  crewmember  interaction  is  minimal,  with  the  result 
that  critical  input  is  ignored  or  not  sought.  Crewmembers  may 
display  one  or  more  of  the  known  hazardous  thought  patterns 
(e.g.,  machoism,  anti-authority,  get-home-itis) .  The  crew  may  be 
unable  to  decide  or  implement  a  course  of  action  before  a 
situation  becomes  critical. 


BASIC  QUALITY  4.  Prioritize  actions  and  distribute  workload 

(Workload) 


Explanation: 

This  is  a  rating  of  the  effectiveness  of  time  and  work 
management.  Rate  the  extent  to  which  the  crew  as  a  team  avoids 
being  distracted  from  essential  activities,  distributes  workload, 
and  avoids  individual  crewmember  overload. 

Exampl es : 

•AH-64  and  UH-60  Task  1080,  Perform  procedures  for  two-way  radio 
failure:  P*  will  remain  focused  outside  the  aircraft  or  inside 
the  cockpit  on  the  instruments,  as  appropriate.  He  will  not 
participate  in  troubleshooting  the  malfunction. 

♦UH-60  Task  2079  and  AH-64  Task  1064,  Perform  terrain  flight 
navigation:  P  will  focus  his  attention  primarily  inside  the 
cockpit;  however,  as  workload  permits,  he  will  assist  in  clearing 
the  aircraft  and  provide  adequate  warning  of  traffic  and 
obstacles . 


Superior  Rating  (7) 

Virtually  all  distractions  are  avoided.  Each  crewmember 
understands  precisely  what  information  is  relevant  to  the  mission 
and  what  information  is  simply  a  distraction.  If  a  crewmember 
becomes  mildly  distracted,  other  crewmembers  remind  him  to  focus 
on  the  mission  task.  Non-critical  duties  are  prioritized  and 
delayed  until  low  workload  periods  or  post-flight  periods. 
Crewmembers  are  aware  of  workload  build  ups  on  others  and 
readjust  workload  by  assuming  emerging,  unassigned  tasks 
appropriate  for  their  duty  station.  Overloads  do  not  occur.  The 
crew's  planning  horizon  is  always  "ahead  of  the  aircraft." 


Acceptable  Rating  (4) 

Most  distractions  are  avoided.  The  crew  performs  well  in 
deciding  what  information  and  activities  are  essential  to  the 
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mission.^  Most  non-6ss6ntial  information  is  discardad  or  ignorad. 
Non-critical  dutias  ara  prioritizad  and  dalayad  until  low 
workload  pariods  or  post-flight  pariods.  Crawmambars  ara  awara 
of  individual  crawmambar  workloads  during  aach  phasa  of  tha 
mission.  Whan  an  individual  crawmambar  appaars  to  ba  ovarloadad, 
othar  crawmambars  taka  on  part  of  tha  workload.  The  crew  is 
always  "in  sync  with  the  aircraft." 


Very  Poor  Rating  (1) 

The  crew  is  easily  distracted.  The  crew  is  unable  or  unwilling 
to  decide  what  is  important  and  relevant  to  the  immediate 
mission.  There  is  little  prioritizing  of  duties  or  actions. 

Time  and  energy  may  be  wasted  on  low  priority  tasks.  Risks  to 
crew  safety  may  occur  as  the  crew  focuses  on  minor  tasks  while 
critical  tasks  requiring  immediate  attention  go  unattended, 

(e.g.,  setting  a  radio  frequency  when  attention  should  be  focused 
on  clearing  an  obstacle.) .  Neither  the  overloaded  party  nor 
other  crewmembers  takes  voluntary  actions  to  eliminate  an 
overload  condition.  The  crew  makes  little  or  no  effort  to 
redistribute  task  responsibilities  as  mission  changes  occur  and 
new  tasks  arise.  Individual  crewmembers  experience  workload 
overloads.  The  crew's  planning  horizon  is  sometimes  "behind  the 
aircraft . " 


BASIC  QUALITY  5,  M^agement  of  unexpected  events  (Unexpt  Events) 


Explanation: 

This  rating  evaluates  the  crew's  performance  under  unusual 
circumstances  that  may  involve  high  levels  of  stress.  This 
judgement  includes  the  integration  of  technical  and  managerial 
aspects  of  contending  with  the  situation.  Note:  Enter  the 
abnormal  or  emergency  situation  in  the  Aircrew  Coordination 
Training  Grade  Slip  (some  emergency  procedure  ATM  tasks  are 
preprinted)  and  grade  it  the  same  as  any  task. 

Examp  1 es : 

•AH-64  and  UH-60  Task  2008,  Perform  evasive  maneuvers:  The  most 
important  consideration  in  an  emergency  is  aircraft  control-- 
first  assess  aircraft  controllability,  check  systems  indicators, 
take  evasive  action. 

♦UH-60  Task  1068,  Perform  or  describe  emergency  procedures:  CE 
will  keep  communications  to  a  minimum  to  allow  the  P*  or  P  to 
attempt  communications  outside  the  aircraft. 
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Superior  Rating  (7) 

The  crew  remains  calm  during  the  situation.  Each  crewmember 
seeks  to  understand  the  problem  and  provides  the  pilot-in-command 
with  essential  information.  Each  crewmember  immediately  takes  on 
particular  workload  responsibilities  based  on  prior  discussions 
and  rehearsal  of  potential  problems  and' contingencies .  The  crew 
effectively  communicates  its  actions  and  results  to  others  and 
provides  feedback  to  ensure  complete  coordination  of  efforts. 

Each  crewmember  handles  his  own  responsibilities  and  seeks  to 
support  the  crewmember  with  the  greatest  workload.  The  crew 
rapidly  imposes  the  maximum  amount  of  control  possible  over  the 
situation  given  the  available  time  and  internal  and  external 
resources.  A  high  level  of  situation  awareness  is  maintained 
throughout  the  event . 


AcceptcUdle  Rating  (4) 

The  crew  responds  to  the  problem  and  the  pilot-in-command's 
requests  for  information  but  does  not  overreact.  The 
pilot-in-command's  requests  for  information  are  met  by  feedback 
from  the  crew.  The  crew  takes  actions  to  reduce  the 
pi lot -in- command ' s  work  overload  and  provides  information  even  if 
it  is  not  specifically  requested.  The  pilot  and  crew  make  good 
use  of  available  resources.  The  crew  is  intense  but  not 
flustered  by  the  situation.  Adequate  situation  awareness  is 
maintained  throughout  the  event . 


Very  Poor  Rating  (1) 

The  crew  becomes  disorganized  and  flustered.  The  pilot-in¬ 
command's  requests  for  information  elicit  inadequate  responses. 
Crewmembers  may  focus  on  the  wrong  issues,  thus  delaying  correct 
diagnosis  of  the  problem.  The  crew  focuses  on  only  one  solution 
to  an  event,  does  not  consider  other  plausible  alternatives,  or 
chooses  an  inappropriate  solution.  Lack  of  coordinated  actions 
adds  to  the  confusion.  The  pilot  and  crewmembers  make  poor  use 
of  available  resources  to  resolve  the  problem.  Situation 
awareness  appears  to  decay  during  the  situation. 


BASIC  QUALITY  6.  Statements  and  directives  clear,  timely, 

relevant,  complete,  and  verified  (Info  xfer) 


Explanation: 

Rate  the  completeness,  timeliness,  and  quality  of  information 
transfer.  Carefully  consider  the  crew's  feedback  techniques  to 
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verify  information  transfer.  In  particular,  evaluate  the  quality 
of  instructions  and  statements  associated  with  navigation 
activities,  obstacle  clearing  activities,  and  instrument 
readouts . 

Examp 1 es : 

•AH-64  Task  1015,  Perform  ground  taxi:  The  P  will  announce 
"Blocking"  to  acknowledge  the  P*'s  announcement  "Braking". 

•UH-60  Task  2079,  Perform  terrain  flight  navigation:  The  P*  will 
acknowledge  commands  issued  by  the  P  for  heading  and  airspeed 
changes . 

Superior  Rating  (7) 

Crewmembers  communicate  frequently.  Both  senders  and  receivers 
use  standard  terminology  for  nearly  all  communications.  Senders 
almost  always  provide  clear,  concise  information.  Receivers 
acknowledge  nearly  all  messages  in  sufficient  detail  so  that  the 
sender  can  verify  that  the  receiver  understands  the  message. 
Receivers  ask  for  clarification  when  they  do  not  understand. 
Senders  pursue  feedback  when  no  response  is  forthcoming.  Whenever 
a  workload  shift  or  task  responsibility  transfer  occurs,  the 
change  is  communicated  and  acknowledged  by  the  crew.  All 
navigation,  obstacle  clearing,  and  "inside"  or  "outside"  the 
cockpit  information  is  stated,  acknowledged,  and  updated. 


Acceptable  Rating  (4) 

Crewmembers  communicate  about  the  mission  as  required.  Standard 
terminology  is  usually  used.  Receivers  acknowledge  most 
messages.  Receivers  ask  questions  when  they  do  not  understand. 
Senders  usually  pursue  feedback  when  no  response  is  forthcoming. 
Crewmembers  are  appraised  of  changes  to  responsibilities  during 
the  flight.  "Inside"  and  "outside"  the  cockpit  duties  are 
specified  and  communicated  to  others. 


Very  Poor  Rating  (1) 

Crewmembers  may  fail  to  make  statements  regarding  critical 
information.  Non-standard  terminology  is  used  or  standard 
terminology  is  used  inappropriately.  Sender  messages  may  be 
inappropriately  delayed  or  irregular  and  may  be  confusing. 
Receivers  usually  do  not  verbally  acknowledge  the  receipt  of 
messages.  Receivers  do  not  ask  questions.  Senders  do  not  pursue 
feedback  when  no  response  is  forthcoming.  Changes  in 
responsibilities  during  the  mission  are  often  not  communicated 
and  may  result  in  confusion  over  who  has  a  task  responsibility. 
Navigation  instructions  and  obstacle  location  information  may  be 
incomplete  or  confusing.  At  times,  "inside"  or  "outside"  the 
cockpit  responsibilities  are  not  clearly  communicated. 
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BASIC  QUALITY  7.  Maintenance  of  miaaion  aituatlon  awareneaa 

(Sit  Aware) 


Explanation: 

This  rating  assesses  the  extent  to  which  crewmembers  keep  each 
other  informed  on  the  status  of  the  aircraft  and  mission 
accomplishment.  This  information  reporting  helps  maintain  a  high 
level  of  situation  awareness  among  the  flight  crew.  Information 
reported  includes : 

•Aircraft  position  and  orientation 
•Equipment  status 
•Personnel  status 

•Environment  and  battlefield  conditions 
•Changes  to  mission  objectives 

Crew-wide  situation  awareness  is  an  essential  element  of  safe 
flying  and  effective  crew  performance. 

Examples: 

•UH-60  Task  2009,  Perform  multi-aircraft  operations:  P  and  CE 
will  provide  adequate  warning  to  avoid  traffic  or  obstacles. 

•AH-64  Task  2008,  Perform  evasive  maneuvers:  When  engaged  by  the 
enemy,  crew  will  announce  the  nature  and  direction  of  the  threat. 

Superior  Rating  (7) 

Crewmembers  routinely  provide  each  other  with  updates  on  the 
status  of  the  elements  of  situation  awareness  and  the  status  of 
the  mission.  Crewmembers  anticipate  the  situation  awareness 
needs  of  others  and  request  needed  information  when  it  is  not 
forthcoming.  Crewmembers  are  aware  of  each  others'  mental  and 
physical  states  and  are  not  hesitant  to  alert  others  to  personal 
problems  that  could  undermine  effective  performance.  Personnel 
status  is  voluntarily  shared  without  fear  of  sanctions.  All 
changes  in  the  elements  of  situation  awareness  are  verbalized  and 
acknowledged.  Crewmembers  alert  other  crewmembers  to  the 
presence  of  obstacles. 


Acceptable  Rating  (4) 

Crewmembers  usually  provide  updates  on  the  status  of  most  of  the 
elements  of  situation  awareness  and  the  status  of  the  mission. 
Changes  to  the  situation  awareness  elements  are  verbalized. 
Obvious  changes  in  personnel  status  are  noted  and  acknowledged 
without  fear  of  sanctions. 
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Very  Poor  Rating  (1) 

Crewmembers  do  not  routinely  provide  updates  on  the  status  of  the 
aircraft  or  the  status  of  the  mission.  Generally,  updates  are 
provided  only  on  request;  they  are  not  made  voluntarily. 

Personnel  problems  such  as  fatigue  or  lack  of  attention  are  not 
mentioned. 


BASIC  QUALITY  8^  Decisions  and  actions  conmunlcated  and 

acknowledged  (Comm/  Ack) 


Explanation: 

Rate  the  extent  to  which  decisions  and  actions  are  actually  made 
and  announced  to  the  crewmembers  after  input  is  solicited  from 
them.  Crewmembers  should  respond  verbally  or  with  the 
appropriate  adjustment  to  their  behaviors,  actions,  or  control 
inputs  to  clearly  indicate  that  they  understand  when  a  decision 
has  been  made  and  what  it  is .  Failure  to  do  so  may  confuse  crews 
and  lead  to  uncoordinated  operation.  Note:  Due  to  time 
constraints  in  certain  situations,  there  is  often  little  or  no 
time  for  crews  to  make  inputs  to  a  decision.  In  such  cases, 
raters  should  focus  on  the  extent  to  which  decisions  are 
acknowledged  verbally  or  through  coordinated,  pre-planned  action. 

Examp  1 es : 

•UH-60  Task  2086,  Perform  masking  and  unmasking:  P*  will  announce 
his  intent  to  unmask.  The  P  and  CE  will  acknowledge  that  they 
are  prepared  to  execute  the  maneuver. 

•AH-64  Task  1038,  Perform  terrain  flight  approach:  P*  will 
announce  intention  of  a  go-around  .  .  .  whether  approach  will 
terminate  to  a  hover  or  to  the  ground.  P  will  acknowledge  use  of 
manual  stabilator  or  any  intent  to  deviate  from  the  approach. 

Superior  Rating  (7) 

The  pi lot -in-command  states  decisions  and  actions  and,  time 
permitting,  explains  the  reasons  and  intent.  Crewmembers 
acknowledge  the  decisions  with  a  clear  verbal  response  and  ask 
questions  to  clarify  any  confusion.  The  pilot -in-command  answers 
all  questions  in  a  positive,  straight-forward  manner. 

Crewmembers  keep  the  pilot -in-command  informed  of  the  results  of 
their  activities  and  changing  responsibilities — especially  visual 
area  of  responsibility  or  task  focus.  The  crew  clearly 
acknowledges  results  of  actions,  or  changes,  and  then  states  its 
intended  adjustments  based  on  the  information  provided.  If 
crewmembers  do  not  acknowledge  or  adjust,  the  pilot-in-command 
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requests  acknowledgement.  Crewmembers  are  particularly  attentive 
to  the  communication  of  workload  responsibilities.  When  assuming 
control  of  the  aircraft  or  making  control  inputs,  notification  is 
always  given  and  acknowledgement  received. 


Acceptable  Rating  (4) 

The  pi lot -in-command  states  decisions  and  actions  along  with, 
time  permitting,  a  brief  explanation  of  the  reasons  and  informs 
the  crew  of  the  adjustments  they  are  expected  to  make.  The  crew 
acknowledges  its  awareness  of  the  decisions  and  directions. 
Crewmembers  may  ask  questions  to  clarify  confusion.  The  pilot 
answers  questions  clearly  and  quickly  and  the  crew  adjusts  to  the 
new  situation.  When  assuming  control  of  the  aircraft  or  making 
control  inputs,  notification  is  given  and  acknowledged. 


Very  Poor  Rating  (1) 

Decisions  and  actions  of  a  crewmember  are  often  not  passed  on  to 
the  crew.  The  pilot-in-command  takes  unilateral  action  and  does 
not  explain  or  inform  the  crew  of  his  intended  purpose.  The  crew 
is  often  not  aware  that  a  decision  has  been  made.  The  crew 
infrequently  asks  questions  for  clarification.  The 
pilot-in-command  may  not  acknowledge  or  respond  to  questions.  The 
crew  may  not  know  how  to  react  to  changed  circumstances. 
Crewmembers  are  often  unsure  what  responsibilities  have  been 
assigned  to  them.  Crewmembers  may  take  uncoordinated  actions 
without  stating  intentions  or  results.  Two  pilots  may  attempt  to 
simultaneously  take  control  of  the  aircraft  when  flight  control 
authority  is  unclear. 


BASIC  QUALITY  9.  Supporting  Information  and  actions  aougbt  from 

crew  (Info  Sought) 


Explanation: 

This  is  a  rating  of  the  extent  to  which  crewmembers,  usually  the 
pilot-in-command,  seek  support  information  and  support  actions 
from  the  crew.  Evaluate  the  degree  to  which  crewmembers  raise 
questions  during  the  flight  regarding  plans,  revisions  to  plans, 
actions  to  be  taken,  and  the  status  of  key  mission  information. 
Note:  The  extent  to  which  crewmembers  maintain  situational 
awareness  and  contribute  to  decision  making  should  be  observed 
here  but  evaluated  on  basic  qualities  7  and  4  respectively. 

Exampl es : 
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•UH-60  Task  1032,  Perform  slope  operations:  P*  will  request 
assistance  in  setting  the  brakes. 

•AH-64  Task  2044,  Perform  actions  on  contact:  The  crew  will 
discuss  options  for  developing  the  situation. 


Superior  Rating  (7) 

During  the  flight,  crewmembers  raise  questions  on  plans  or 
changes  to  plans  and  actions.  Virtually  all  of  these  inquiries 
surface  information  that  contributes  to  the  mission  decision 
making  process.  When  the  pilot-in-command  realizes  that  a 
decision  must  be  made  during  the  flight,  for  which  there  is  no 
clear  standardized  answer,  he  immediately  alerts  the  crew  to  the 
situation  and  seeks  suggestions  on  possible  solutions  and 
important  information  to  consider.  The  pilot-in-command  is  open 
to  all  suggestions.  Crewmembers  respond  to  these  inquiries  with 
sound,  task-focused  discussions  and  clear  answers  that  are 
provided  in  a  timely  manner.  Crewmembers'  inquiries  are  never 
ignored.  All  crewmembers  encourage  such  questioning.  When  the 
pilot-in-command  asks  for  assistance  with  actions  he  clearly 
states  what  assistance  is  required.  He  provides  quick,  clear 
feedback  if  the  crewmember  response  is  not  what  he  expects.  He 
asks  for  assistance  before  becoming  overloaded. 


Acceptable  Rating  (4) 

During  the  flight,  crewmembers  occasionally  raise  questions  on 
plans  or  actions  when  they  are  unclear  on  decisions  being  made. 
Most  of  these  inquiries  provide  information  that  is  relevant  to 
the  mission  decision  making  process.  The  pilot  alerts  the  crew 
to  the  need  for  decision  input.  Crewmembers  usually  respond  to 
these  inquiries  with  brief  but  reasonable  answers.  Crewmembers' 
inquiries  are  encouraged  by  other  crewmembers  most  of  the  time. 
The  pilot-in-command  listens  to  suggestions  without  interruption 
or  criticism.  He  asks  for  clarification  as  necessary.  He  only 
asks  for  assistance  when  he  becomes  overloaded. 


Very  Poor  Rating  (1) 

During  the  flight,  crewmembers  almost  never  raise  questions  about 
plans,  actions,  or  changes  to  plans.  The  pilot -in-command  makes 
mission  decisions  without  seeking  inputs  from  other  crewmembers. 
The  pilot-in-command  does  not  alert  the  crew  that  a  decision  is 
required  or  is  being  made.  Decision  making  and  planning  are  done 
by  one  individual  with  little  or  no  discussion--an  observer  will 
have  difficulty  noting  this  quality  for  "very  poor"  crews  since 
it  is  hard  to  detect  individual  decision  making.  The  few 
inquiries  that  are  made  are  generally  ignored  or  abruptly 
answered.  Crewmembers  may  discourage  others  from  asking 
questions  by  the  tone  of  voice  they  use  or  by  failing  to  respond. 
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The  pilot -in-command  may  not  ask  for  crew  assistance  with  tasks 
even  when  he  is  overloaded  to  the  point  of  nearly  failing  to 
properly  execute  tasks . 


BASIC  QUALITY  10.  Cretmember  actions  mutually  cross  monitored 

(Cross  Monitor) 


Explanation: 

This  rating  captures  the  extent  to  which  a  crew  uses  cross 
monitoring  as  a  mechanism  to  avoid  errors  and  improve  future 
performance.  Crewmembers  are  able  to  catch  each  other's  errors. 
Such  redundancy  is  particularly  important  when  crews  are  fatigued 
or  overly  focused  on  critical  task  elements,  and  thus  more  prone 
to  make  errors.  Included  in  this  rating  is  the  crew's  use  of 
aircraft  technical  manual  checklists  to  perform  required 
procedure  checks  and  procedures  (i.e.,  engine-start,  run-up, 
before-takeoff,  before-  and  after-landing,  shutdown  checks;  HIT 
and  emergency  procedures) .  Note:  This  quality  does  not  imply 
that  task  responsibilities  are  not  clearly  defined.  It  asks  the 
question  "To  what  extent  do  crewmembers  help  an  individual 
assigned  primary  responsibility  for  a  task  or  action  by  reviewing 
the  quality  of  that  individual's  task  execution  and  alerting  him 
to  any  mistake  noted?" 

Examples: 

•AH-64  Task  1094,  Identify  major  US  or  allied  equipment  and  major 
threat  equipment:  P*  or  P  will  announce  the  type  and  direction  of 
the  equipment  detected.  The  other  crewmember  will  confirm  the 
type  and  direction  of  the  equipment. 

•UH-60  task  1023,  Perform  fuel  management  procedures:  PC  will 
confirm  the  results  of  the  fuel  check. 

Superior  Rating  (7) 

Each  crewmember  is  concerned  that  all  tasks  are  properly  executed 
and  checks  both  his  tasks  and  those  of  others.  When  mistakes  are 
noted,  the  crewmember  making  the  error  is  quickly  informed  in  a 
concise  manner  without  excessive  formality.  The  mistake  maker 
accepts  this  review  and  feedback  as  a  normal  part  of  crew 
operations . 


Acceptable  Rating  (4) 

Crewmembers  often  check  each  other's  task  performance  for  errors. 
Mistake  makers  are  informed  and  make  the  needed  corrections.  Only 
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occasionally  are  mistake  makers  annoyed  at  being  checked  and 
corrected. 


Very  Poor  Rating  (1) 

Crewmembers  seldom,  if  ever,  check  each  other's  task  execution. 
Crewmembers  are  insulted  if  they  are  corrected  by  another 
crewmember . 


BASIC  QUALITY  11,  Supporting  Information  and  actions  offered 

by  crew  (info  Offered) 


Explanation: 

This  is  a  rating  of  the  extent  to  which  crewmembers  anticipate 
and  offer  support  information  and  support  actions  to  the  decision 
maker,  usually  the  pilot-in-command,  when  it  becomes  apparent 
that  a  decision  must  be  made  or  an  action  taken. 


Exampl es : 

•UH-60  Task  2016,  Perform  external  load  operations:  All 
crewmembers  will  assist  in  clearing  the  aircraft  and  will  provide 
adequate  warning  of  obstacles,  unusual  drift,  or  altitude 
changes . 

♦UH-60  and  AH-64  Task  1081,  Perform  nonprecision  approach:  P  will 
call  out  the  approach  procedure  to  the  P* . 

Superior  Rating  (7) 

The  crew  recognizes  that  a  decision  must  be  made  and  offers 
suggestions  and  information  to  the  pilot-in-command.  The  crew 
checks  for  responses  that  indicate  understanding.  The 
information  is  repeated,  as  necessary,  to  ensure  that  the  pilot- 
in-command  understands  the  input.  Pilot -in-command  responses  can 
be  verbal  or  non-verbal  actions .  The  crew  seeks  information  and 
provides  it  to  support  decisions  and  actions.  The  crew 
frequently  offers  task  execution  support.  The  support  offered 
always  reflects  the  pilot-in-command's  task  needs.  Crews  are 
quick  to  offer  support  during  particularly  difficult  tasks  such 
as  obstacle  clearing. 
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Acceptable  Rating  (4) 

The  crew  recognizes  that  a  decision  or  action  must  be  made  and 
offers  suggestions  and  information  to  the  pilot-in-command.  The 
crew  sometimes  offers  task  execution  support.  Crewmembers 
usually  offer  obstacle  clearing  support. 


Very  Poor  Rating  (1) 

The  crew  does  not  offer  suggestions  and  inputs  to  support 
decision  making  or  actions.  Moreover,  it  often  appears  that  the 
crew  does  not  even  realize  that  a  decision  is  being  made.  The 
crew  generally  does  not  offer  its  services  to  support  task 
execution  for  other  crewmembers.  Crewmembers  may  fail  to  offer 
obstacle  clearing  support. 


BASIC  QUALITY  12.  Advocacy  and  assertion  practiced 

(Advoc /Assert ) 


Explanation: 

This  rating  evaluates  the  extent  to  which  crewmembers  advocate  a 
course  of  action  they  consider  best,  even  when  it  may  differ  with 
the  one  being  followed  or  proposed.  Note:  Except  under  extreme 
emergency  conditions  where  time  is  absolutely 

critical,  it  is  usually  in  the  crew's  best  interest  to  hear  the 
full  range  of  viewpoints  available. 

Exampl es : 

•UH-60  and  AH-64  Task  2083,  Negotiate  wire  obstacles:  Crew  will 
discuss  the  characteristics  of  the  wires  ...  to  determine  the 
method  of  crossing. 

•AH-64  Task  2044,  Perform  actions  on  contact:  Crew  will  discuss 
options  for  developing  the  situation. 

Superior  Rating  (7) 

Crewmembers  state  to  the  rest  of  the  crew  a  course  of  action  that 
they  consider  best.  They  clearly  explain  their  reasons  for 
believing  this  to  be  the  best  course.  Other  crewmembers  listen 
to  the  argument  before  presenting  any  criticism  or  proposing 
alternate  courses.  Discussions  focus  on  the  strengths  and 
weaknesses  of  the  proposed  course  of  action,  not  on  the 
personality  of  the  crewmember  who  proposed  the  action. 
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Crewmembers  call  the  crew's  attention  to  changes  in  the  situation 
and  provide  information  that  is  essential  to  the  proper  execution 
of  another  crewmember's  task.  Crewmembers  pursue  feedback  to 
ensure  that  their  views  are  heard  and  understood.  Other 
crewmembers  expect  such  open  comments  and  view  them  as  positive 
contributions  to  mission  performance. 


Acceptable  Rating  (4) 

Crewmembers  state  their  support  for  a  course  of  action  or  suggest 
improvements  to  other  proposed  actions .  Each  crewmember  makes  an 
effort  to  explain  his  position  and  convince  others  to  concur  with 
him  on  the  course  of  action  to  be  taken.  Other  crewmembers  may 
interrupt  with  their  views  and  alternatives.  Crewmembers  usually 
speak  out  when  they  recognize  a  departure  from  the  mission  plan 
or  standard  procedures  or  when  they  have  a  piece  of  information 
that  is  important  to  another's  task  execution.  Crewmembers  seek 
assurances  that  presented  information  has  been  received.  Other 
crewmembers  view  such  comments  as  constructive  and  not  as  a 
challenge  to  authority. 


Very  Poor  Rating  (1) 

The  crew  almost  never  suggests  a  course  of  action.  Crewmembers 
attempting  to  propose  a  course  of  action  may  be  cut-off  before 
they  can  propose  the  action  or  explain  the  rationale  for  that 
action.  Crewmembers  proposing  courses  of  action  may  receive 
personal  attacks.  The  crew  raises  few,  if  any  concerns. 
Crewmembers  may  even  fail  to  intervene  when  risks  such  as 
obstacles  or  poor  visibility  arise. 


BASIC  QUALITY  13,  Crew- level  after-action  reviews  accomplished 

(AAR) 


Explanation: 

This  rating  evaluates  the  extent  to  which  the  crew  reviews  and 
critiques  its  decisions  and  actions  during  or  following  a  mission 
segment,  during  low  workload  periods,  or  during  the  post  flight 
debrief.  Evaluate  the  crew  on  their  discussion  of  strengths  and 
weaknesses  (for  example,  what  was  done  wrong,  what  might  be  done 
better,  how  improvements  can  be  made,  and  what  was  done  very 
well)  in  flight  skills  and  aircrew  coordination. 

Superior  Rating  (7) 

The  entire  crew  reviews  and  critiques  its  decisions  and  actions 
throughout  the  mission,  including  the  pre-mission  planning  and 
rehearsal  process.  Crewmembers  review  factors  considered  in 
making  their  decisions,  identify  additional  options  or  factors. 
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including  ways  to  "buy  time, "  that  should  have  been  considered, 
and  discuss  different  methods  of  weighing  information  in  the 
decision  process.  All  discussions  focus  on  behaviors  and 
information  and  carefully  avoid  any  "finger-pointing"  tones.  The 
focus  is  clearly  on  education  and  understanding  to  improve 
individual  and  collective  performance. 


Acceptable  Rating  (4) 

Senior  crewmember ( s )  review  and  critique  the  crew's  decisions  and 
actions  during  problematic  segments  of  the  mission.  They 
determine  the  major  mistakes  in  the  crew's  actions  or  decisions 
and  identify  remedial  actions  or  alternative  options  for  future 
missions.  Although  the  critiques  are  intended  to  educate  the 
crew  and  to  improve  their  performance  during  future  missions, 
they  may  include  some  accountability  for  unsatisfactory 
performance . 


Very  Poor  Rating  (1) 

The  crew  either  fails  to  review  and  critique  its  mission 
performance  or  if  a  critique  is  performed,  it  is  punitive  or 
accusatory.  That  is,  the  critique  is  conducted  primarily  to 
assign  blame  for  unsatisfactory  performance.  Little  effort  is 
made  to  identify  lessons  learned  or  to  suggest  constructive  ways 
to  improve  future  performance. 
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Aircrew  Coordination  Basic  Qualities  and  Rating  Factors 
Basic  Quality  1  Establish  and  maintain  flight  team  leadership  and  crew  donate 
Issue  1-1  Leadership  Style 

a.  What  type  of  cockpit  atmosphere  did  the  PC  establish? 

□  PC  actively  established  an  open  climate  where  crew  members  freely  talked  and  asked 
questions 

o  PC  permitted  an  open  climate  with  some  discussion  and  questioning  among  crew  members 
a  PC  created  a  restrictive  climate  by  means  of  an  authoritarian  management  style 

b.  Did  the  PC  acknowledge  each  individual  to  be  a  part  of  the  crew? 

□  PC  considered  each  crew  member  to  be  an  important  contributor  to  mission  success 
a  PC  acknowledged  each  crew  member  as  part  of  the  team 

□  Some  crew  members  felt  ignored  by  the  PC  and  were  reluctant  to  speak  up 

Issue  1-2  Professionai  Respect 

a.  Did  crew  members  show  professional  respect  to  each  other? 

□  Each  crew  member  was  valued  for  their  expertise  and  judgement 

□  Crew  members  showed  an  acceptable  level  of  professional  courtesy  to  one  another 
o  Crew  members  openly  or  indirectly  belittled  one  another 

b.  Did  rank  or  experience  differences  influence  the  respect  shown  to  junior  crew  members? 

n  Junior  crew  member  performance  was  actively  promoted  through  positive  encouragement 
and  professional  respect 

□  Junior  crew  members  were  shown  professional  respect  regardless  of  rank  or  flight  hour 
experience 

n  Junior  crew  members  were  shown  little  or  no  professional  respect  because  of  their  rank  or 
flight  hour  experience 

Issue  1-3  Resolution  of  Disagreements 

a.  Were  differences  of  opinion  appropriately  surfaced  between  crew  members? 

□  Alternative  viewpoints  were  considered  a  normal  and  occasional  part  of  crew  interaction 
Q  Some  alternative  viewpoints  were  tolerated  and  did  not  lead  to  obvious  disruption-  of 

teamwork 

□  Disagreements  existed  among  the  crew  members,  but  were  rarely  surfaced  for  resolution 
d.  How  were  conflicts  handled  between  crew  members? 

□  Disagreements  were  handled  in  a  professional  manner  without  involving  personal  attacks  or 
defensive  posturing 

□  Disagreements  did  not  involve  obvious  attacks  of  character  or  defensive  posturing 

□  Conflicts  involved  personal  attacks  and  resulted  in  a  disruption  of  teamwork 
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Issue  1*4  Crew  Member  Attitudes 

a.  Did  the  PC  play  an  active  role  in  managing  crew  attitudes? 

o  PC  actively  monitored  the  attitudes  of  crew  members  and  offered  feedback  when  necessary 

a  PC  took  steps  to  correct  obvious  displays  of  improper  or  hazardous  attitudes 

□  PC  exhibited  a  hazardous  attitude,  or  tolerated  such  an  attitude  in  other  crew  members 

b.  Did  the  crew  members  give  proper  consideration  to  safety? 

a  Each  crew  member  actively  displayed  a  proper  concern  for  balancing  safety  with  mission 
accomplishment 

□  Crew  members  did  not  display  any  obvious  disregard  for  safety  during  the  mission 

n  Displays  of  hazardous  attitudes  by  one  or  more  crew  members  potentially  jeopardized  flight 

safety 

Basic  Quality  2  Premission  planning  and  rehearsal  accomplished 

Issue  2-1  Premission  Flight  Planning 

a.  Did  the  PC  assign  critical  actions,  duties,  and  mission  responsibilities? 

□  PC  actively  insured  that  all  actions,  duties,  and  mission  responsibilities  were  partitioned  and 
clearly  assigned  to  specific  crew  members 

□  No  obvious  assignments  were  overlooked  regarding  critical  actions,  duties,  or  mission 
responsibilities 

□  One  or  more  critical  actions,  duties,  or  mission  responsibilities  were  overlooked  during  the 
premission  planning  process 

b.  How  involved  was  each  crew  member  in  the  planning  process? 

□  Each  crew  member  was  actively  involved  in  the  mission  planning  process  so  as  to  insure  a 
common  understanding  of  mission  intent  and  operational  sequence 

□  All  crew  members  had  a  general  understanding  of  mission  intent  and  operational  sequence; 
each  crew  member  participated  at  least  minimally  in  the  planning  process 

□  One  or  more  crew  members  had  an  inadequate  or  incorrect  understanding  of  mission  intent 
or  operational  sequence  due  to  lack  of  involvement  in  planning  process 

c.  Did  the  crew  prioritize  its  planning  activities  within  the  available  time? 

□  Planning  activities  were  prioritized  to  insure  that  critical  items  were  addressed  within  the 
available  planning  time 

□  Planning  activities  were  generally  prioritized  within  the  available  time,  with  no  major 
component  of  the  mission  overlooked 

□  Little  attention  was  given  to  prioritizing  planning  activities  within  the  available  planning  time; 
some  components  of  the  mission  were  neglected 
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Issue  2-2  Ptemission  Rehearsal 

a.  Were  alternative  courses  of  action  identified  and  discussed? 

o  Alternative  courses  of  action  were  identified  in  anticipation  of  potential  changes  in  METT-T 
factors;  crew  was  fully  prepared  to  implement  contingencies  as  necessary 

a  Some  attention  was  given  to  identifying  alternative  courses  of  action;  crew  reacted  to 
changes  in  METT-T  factors,  but  some  additional  planning  was  required 

□  Little  or  no  attention  was  paid  to  identifying  alternative  courses  of  action;  changes  in  METT-T 
factors  required  substantial  replanning  by  crew  members 

b.  Did  the  crew  members  mentally  rehearse  critical  flight  segments? 

Q  Crew  members  mentally  rehearsed  the  entire  mission  by  visualizing  and  discussing  potential 
problems,  contingencies,  and  responsibilities 

□  Crew  members  gave  some  attention  to  discussing  potential  problems,  contingencies,  and 
responsibilities;  however,  they  stopped  short  of  mentally  rehearsing  critical  flight  segments  or 
sequences 

□  Crew  failed  to  discuss  or  rehearse  critical  flight  segments  and  sequences;  they  began  the 
mission  with  very  limited  understanding  or  agreement  concerning  potential  problems, 
contingencies,  or  responsibilities 

Issue  2*3  In-Flight  Replanning  and  Rehearsal 

a.  Did  the  crew  take  advantage  of  low  workload  periods  to  rehearse  up-coming  flight  segments? 

n  PC  actively  insured  that  crew  members  took  advantage  of  low  workload  periods  during 
mission  to  rehearse  upcoming  flight  segments 

o  Crew  members  engaged  in  some  in-flight  rehearsal  of  up-coming  flight  segments;  no  major 
coordination  problems  arose  that  could  be  attributed  to  a  failure  to  rehearse 

□  Little  or  no  attention  was  given  to  in-flight  rehearsal  of  up-coming  flight  segments;  some 
coordination  problems  were  attributed  to  a  failure  to  rehearse 

b.  Did  the  crew  anticipate  required  adjustments,  keeping  ahead  of  critical  time  horizons? 

□  Crew  members  continuously  reviewed  remaining  flight  segments  to  identify  required 
adjustments;  planning  consistently  kept  ahead  of  critical  time  horizons 

□  Crew  members  occasionally  reviewed  remaining  flight  segments  to  identify  required 
adjustments;  planning  generally  kept  ahead  of  critical  time  horizons 

o  Through  inattention,  crew  members  frequently  fell  behind  in  anticipating  required  adjustments 
as  the  mission  progressed;  in-flight  planning  consistently  appeared  rushed 

Basic  Quality  3  Selection  of  appropriate  decision  making  techniques 

Issue  3>1  High  Time  Stressed  Decisions 

a.  Did  the  crew  avoid  excessive  deliberation  when  inconsistent  with  time  urgency  of  decision? 

□  Crew  members  consistently  relied  on  a  pattern-recognition  decision  process  to  produce 
timely  responses;  deliberation  is  minimized,  consistent  with  available  decision  time 

□  Crew  members  generally  avoided  excessive  deliberation  when  it  was  inconsistent  with  the 
time  urgency  of  the  decision;  decisions  met  minimal  time  requirements 

□  Crew  members  displayed  an  inflexible  decision  style,  engaging  in  excessive  deliberation; 
delayed  decisions  frequently  compounded  the  difficulties  faced  by  the  aircrew 
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b.  Did  the  crew  members  focus  on  only  the  most  critical  factors  influencing  the  decision? 

a  Crew  members  consistently  displayed  an  ability  to  focus  on  only  the  most  critical  factors 
influencing  their  choice  of  response 

□  Crew  members  generally  addressed  the  most  critical  factors,  while  occasionally  being 
distracted  by  secondary  issues 

o  Crew  members  lost  focus  and  became  distracted  by  secondary  factors 

c.  Did  the  crew  members  efficiently  prioritize  their  specific  information  needs? 

□  Crew  members  efficiently  prioritized  their  specific  information  needs,  consistent  with  available 
decision  time 

□  Crew  members  generally  requested  only  information  that  was  obtainable  within  the  available 
decision  time 

a  Crew  members  delayed  critical  decisions  by  seeking  information  that  was  unattainable  within 
the  available  decision  time 

Issue  3-2  Moderate/Low  Time  Stressed  Decisions 

a.  Did  crew  members  adequately  deliberate  each  decision,  consistent  with  available  time? 

□  Crew  members  consistently  relied  on  an  analytical  decision  process  to  produce  high  quality 
decisions;  deliberation  was  encouraged,  consistent  with  available  decision  time 

D  Crew  members  generally  engaged  in  some  deliberation,  avoiding  decisions  that  appeared  to 
reflect  impulsive  or  hazardous  attitudes 

o  Crew  members  displayed  an  inflexible  decision  style,  frequently  responding  in  an  impulsive 
manner  with  little  or  no  deliberation:  impulsive  decisions  frequently  compounded  the 
difficulties  faced  by  the  crew 

b.  Did  crew  members  address  each  important  factor  influencing  the  decision? 

□  Crew  members  consistently  considered  all  important  factors  influencing  their  choice  of 
action,  seeking  the  most  unbiased  decision  possible 

□  Crew  members  generally  addressed  the  important  factors  influencing  their  choice  of  action, 
avoiding  obvious  decision  biases  or  gaps  in  logic 

□  Crew  members  overlooked  one  or  more  important  factors  influencing  their  choice  of  action; 
one  or  more  types  of  decision  biases  were  evident  in  their  thinking 

c.  Did  crew  members  seek  out  all  available  information? 

□  Crew  members  consistently  sought  out  all  available  information  relative  to  the  factors  being 
considered 

□  Crew  members  generally  sought  out  the  most  important  information  relative  to  the  factors 
being  considered 

o  Crew  members  failed  to  seek  out  one  or  more  important  pieces  of  available  information 
relative  to  the  factors  being  considered 
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Basic  Quality  4  Prioritize  actions  and  distribute  workioad 

Issue  4-1  Task  Prioritization 

a.  Did  the  crew  consistently  prioritize  competing  mission  tasks? 

n  Crew  members  were  consistently  able  to  identify  and  prioritize  competing  mission  tasks; 
flight  safety  and  other  high-priority  tasks  were  never  ignored;  low-priority  tasks  were 
appropriately  delayed  until  they  do  not  compete  with  more  critical  tasks 

a  Crew  members  were  generally  able  to  maintain  a  focus  on  flight  safety  and  other  high- 
priority  mission  tasks;  task  prioritization  was  acceptable,  but  could  be  improved 

□  Low-priority  tasks  were  occasionally  attended  to  at  the  expense  of  flight  safety  or  other 
higher  priority  tasks;  significant  compromises  in  flight  safety  or  mission  effectiveness 
occurred 

b.  Were  the  crew  members  distracted  by  non-essentiai  events  or  radio  traffic? 

□  Crew  members  consistently  avoided  non-essential  distractions;  distractions  had  no  impact  on 
task  performance 

□  Crew  members  generally  avoided  non-essential  distractions;  some  distractions  arose,  but 
had  no  impact  on  flight  safety  or  mission  effectiveness 

a  Crew  members  were  easily  distracted  by  non-essential  events  and  radio  traffic;  distractions 
resulted  in  significant  compromises  in  flight  safety  or  mission  effectiveness 

Issue  4-2  Woikioad  Distribution 

a.  Did  the  PC  appropriate  manage  the  distribution  of  mission  tasks  to  prevent  overloads? 

□  PC  actively  managed  distribution  of  mission  tasks  to  prevent  any  crew  member  from  being 
task  overloaded,  especially  during  critical  phases  of  flight 

□  Distribution  of  workload  was  not  optimal,  but  no  serious  incidents  of  task  overload  occurred 
with  any  one  crew  member 

□  Maldistribution  of  workload  occurred;  task  overload  of  one  or  more  crew  members 
significantly  related  to  an  issue  of  flight  safety  or  mission  effectiveness 

b.  Did  the  crew  members  cooperatively  adjust  individual  task  responsibilities  to  prevent  overload? 

□  Crew  members  were  consistently  aware  of  workload  buildup  on  others  and  reacted  quickly  to 
adjust  distribution  of  task  responsibilities 

□  Crew  members  maintained  some  awareness  of  workload  buildup  on  others;  workload  was 
adjusted  before  serious  compromise  to  flight  safety  or  mission  effectiveness  occurs 

□  Crew  members  were  generally  unaware  of  workload  buildup  on  others;  little  or  no  attempt 
was  made  to  adjust  the  distribution  of  task  responsibilities  before  significant  compromises  to 
flight  safety  or  mission  effectiveness  occurred 

Basic  Quality  5  Management  of  unexpected  events 

Issue  5-1  Crew  Preparation  and  Composure 

a.  Did  the  crew  reflect  a  consistent  understanding  of  emergency  procedures? 

o  Crew  actions  reflected  extensive  rehearsal  of  emergency  procedures  in  prior  training  and 
premission  planning  and  rehearsal 
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□  Crew  actions  reflected  consistent  understanding  of  emergency  procedures:  responses  were 
adequately  standardized  to  avoid  significant  conflicts  or  misunderstandings 

a  Crew  actions  reflected  misunderstanding  of  emergency  procedures;  little  or  no  evidence  of 
prior  rehearsal  during  training  or  premission  planning  was  evident 

b.  Did  the  crew  react  in  a  coordinated  manner  with  minimal  verbal  direction? 

□  Crew  member  actions  and  information  exchange  were  highly  coordinated  with  minimal  verbal 
direction  from  the  PC 

□  Crew  member  actions  and  information  exchange  proceeded  smoothly,  although  moderate 
direction  from  the  PC  was  necessary 

□  Crew  actions  and  information  exchange  required  extensive  direction  from  the  PC  in  order  to 
avoid  significant  conflicts  or  misunderstandings 

c.  Did  the  crew  display  a  calm,  professional  composure? 

□  Crew  members  responded  in  a  composed,  professional  manner 

a  Crew  composure  was  tense,  but  not  flustered 

□  Crew  composure  was  disorganized  and  flustered 

Issue  5*2  Resource  Management 

a.  Did  crew  members  adjust  workload  and  task  priorities  with  minimal  verbal  direction? 

□  Each  crew  member  appropriately  adjusted  individual  workload  and  task  priorities  with 
minimal  verbal  direction  from  the  PC 

□  Each  crew  member  appropriately  adjusted  individual  workload  and  task  priorities,  although 
moderate  direction  from  the  PC  was  necessary 

□  One  or  more  crew  members  failed  to  appropriately  adjust  workload  during  the  course  of  the 
unexpected  event,  resulting  in  a  significant  compromise  to  flight  safety  or  mission 
effectiveness 

b.  Was  each  crew  member  effectively  utilized  in  responding  to  the  unexpected  event? 

□  Each  crew  member  was  effectively  utilized  in  responding  to  the  emergency:  workload  was 
efficiently  distributed 

□  Each  crew  member  was  utilized  in  responding  to  the  emergency,  with  no  major 
maldistributions  of  workload 

□  One  or  more  crew  members  was  inappropriately  utilized  or  underutilized,  resulting  in  a 
significant  compromise  to  flight  safety  or  mission  effectiveness:  other  crew  members 
experienced  task  overload 

Basic  Quality  6  Statements  and  directives  are  clear,  timely,  relevant,  complete,  and  verified 

Issue  6-1  Adequacy  and  timeliness 

a.  Were  all  required  or  recommended  call-outs  accomplished  by  crew  members? 

□  All  required  or  recommended  call-outs  (as  defined  in  the  ATM)  were  made  by  crew  members 
on  a  consistent  basis 

□  Required  or  recommended  call-outs  were  generally  made  by  crew  members,  with  no  major 
omissions  that  significantly  compromised  flight  safety  or  mission  effectiveness 

□  Required  or  recommended  call-outs  were  frequently  ignored  by  crew  members:  significant 
compromises  in  flight  safety  or  mission  effectiveness  occurred 
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b.  Were  statements  and  directives  provided  in  a  timely  manner? 

a  Statements  and  directives  were  consistently  offered  in  a  timely  manner 

□  Statements  and  directives  were  generally  offered  In  a  timely  manner,  with  no  delays  that 
significantly  compromised  flight  safety  or  mission  effectiveness 

o  Statements  and  directives  were  frequently  late,  creating  additional  workload  pressure  for 
other  crew  members:  significant  compromises  to  flight  safety  or  mission  effectiveness 
occurred 

Issue  6-2  Clarity 

a.  Did  the  crew  use  standard  terminology? 

□  Crew  members  consistently  used  standard  terminology  for  ail  communications 

□  Crew  members  generally  used  standard  terminology:  no  significant  misunderstandings 
occurred 

□  Crew  members  failed  to  use  standard  terminology,  resulting  in  misunderstandings 

b.  Were  statements  and  directives  clear  and  concise? 

□  Statements  and  directives  were  clear  and  concise 

□  Statements  and  directives  communicated  clear  messages:  ambiguity  was  generally  avoided 

□  Statements  and  directives  contained  ambiguous  references  or  irrelevant  information 

Issue  6-3  Acknowledgement 

a.  Did  crew  members  seek  feedback  when  misunderstandings  were  apparent? 

□  Crew  members  actively  sought  feedback  when  there  was  no  acknowledgement  from  another 
crew  member 

□  Crew  members  sought  feedback  when  it  appeared  that  another  crew  member  has 
misunderstood  a  statement  or  directive 

□  Crew  members  disregarded  the  need  for  feedback:  misunderstandings  occurred  between 
crew  members 

b.  Did  crew  members  acknowledge  statements  and  directives  from  others? 

□  Crew  members  consistently  acknowledged  all  messages  to  indicate  their  understanding  of 
intent:  crew  members  consistently  requested  clarification  when  necessary 

□  Crew  members  generally  acknowledged  messages  and  requested  clarification  when 
necessary;  no  significant  misunderstandings  occurred 

o  Crew  members  frequently  failed  to  acknowledge  messages;  misunderstandings  occurred 

Basic  Quality  7  Maintenance  of  mission  situational  awareness 

Issue  7-1  Awareness  Level  of  Crew 

a.  Did  crew  members  keep  each  other  aware  of  critical  mission  factors,  obstacles,  or  conditions? 

□  Crew  members  routinely  updated  each  other  on  critical  mission  factors,  obstacles,  or 
conditions;  significant  changes  were  highlighted  and  acknowledged 

□  Crew  members  occasionally  updated  each  other  on  critical  mission  factors,  obstacles,  or 
conditions:  no  significant  compromises  to  flight  safety  or  mission  effectiveness  occurred 
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a  Crew  members  disregarded  need  to  keep  each  other  informed  of  critical  mission  factors, 
obstacles,  or  conditions:  significant  compromises  to  flight  safety  or  mission  effectiveness 
occurred  as  a  result 

b.  Did  crew  members  take  responsibility  for  scanning  entire  flight  environment,  consistent  with  their 

primary  duties? 

□  Crew  members  took  personal  responsibility  for  scanning  the  entire  flight  environment, 
consistent  with  performing  their  primary  cockpit  duties 

□  Crew  members  took  personal  responsibility  for  notifying  other  crew  members  of  significant 
changes,  even  when  they  occurred  within  someone  else's  area  of  scanning  responsibility 

Q  Crew  members  confined  their  attention  exclusively  to  their  assigned  area  of  scanning 
responsibility 

Issue  7*2  Awareness  of  Factors  Inhibiting  Attention 

a.  Were  crew  members  aware  of  the  factors  that  can  degrade  situational  awareness? 

□  Crew  members  actively  discussed  conditions  and  situations  that  could  compromise 
situational  awareness  during  the  mission  (e.g.,  stress,  boredom,  fatigue,  anger) 

a  Crew  members  adjusted  scanning  and  reporting  patterns  in  accordance  with  the  changing 
demands  of  the  mission 

□  Crew  members  appeared  unaware  of  factors  that  could  compromise  situational  awareness; 
stress,  boredom,  fatigue,  and/or  anger  were  seen  to  affect  crew  member  scanning 

Basic  Quality  8  Decisions  and  actions  communicated  and  actowwiedged 

Issue  8-1  Communication  of  Decisions  and  Actions 

a.  Were  decisions  and  actions  announced  in  timely  manner,  with  appropriate  rationale? 

□  Crew  members  announced  decisions  and  actions,  providing  rationale  and  intentions  as  time 
permits:  verbal  communications  were  clear  and  timely 

a  Crew  members  verbally  announced  decisions  and  actions;  instances  of  significant  confusion 
and  surprise  were  avoided 

□  Crew  members  failed  to  announce  decisions  and  actions;  decisions  and  actions  were 
communicated  only  by  ambiguous  body  signals  or  utterances;  confusion  or  surprise  existed 
among  crew  members  regarding  actions  or  decisions  taken 

b.  Did  crew  members  accomplish  transfer  of  controls  in  a  positive  manner? 

n  Crew  members  verbally  coordinated  transfer  of  controls  or  control  inputs  before  initiating 
action:  transfer  of  controls  occurred  smoothly 

□  Crew  members  verbally  coordinated  transfer  of  controls  or  control  inputs  before  initiating 
action;  transfer  of  controls  occurred  without  significant  compromise  to  flight  safety 

□  Crew  members  made  unannounced  control  inputs;  significant  compromise  to  flight  safety 
occurred 
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Issue  8-2  Clarification  and  Acknowledgement 


a.  Did  crew  members  acknowledge  announced  decisions  or  actions,  noting  impact  on  other  crew 

tasks  and  responsibilities? 

□  Crew  members  consistently  acknowledged  announced  decisions  or  actions,  providing 
appropriate  feedback  regarding  the  impact  on  other  crew  tasks  and  responsibilities 

n  Crew  members  generally  acknowledged  announced  decisions  or  actions:  instances  of 
significant  confusion  were  avoided 

□  Crew  members  frequently  failed  to  acknowledge  announced  decisions  or  actions;  confusion 
arose  among  the  crew  members 

b.  Did  crew  members  request  clarification  of  decisions  and  actions,  if  not  understood? 

□  Crew  members  promptly  requested  clarification  of  decisions  or  actions,  as  appropriate 

n  Crew  members  requested  clarification  of  decisions  or  actions  if  significant  confusion  arose 

□  Crew  members  ignored  need  to  clarify  actions  or  decisions,  resulting  in  significant  instances 
of  surprise  and  confusion;  compromises  to  flight  safety  or  mission  effectiveness  occurred 

Basic  Quality  9  Supporting  Mbmiation  and  actions  sought  from  crew 

Issue  9-1  Solicitation  of  Crew  Input 

a.  Did  the  PC  solicit  input  from  other  crew  members  regarding  issues  affecting  flight  safety  or 

mission  effectiveness? 

□  Crew  members  were  encouraged  by  PC  to  actively  raise  issues  or  offer  information  affecting 
flight  safety  or  mission  effectiveness 

□  PC  occasionally  permitted  crew  members  to  raise  issues  affecting  flight  safety  or  mission 
effectiveness;  no  significant  compromises  to  flight  safety  or  mission  effectiveness  occurred 

□  PC  ignored  crew  input  or  gave  the  impression  that  such  inputs  were  unwelcome;  significant 
compromises  to  flight  safety  or  mission  effectiveness  arose  when  crew  inputs  were  ignored 
or  stifled  by  PC 

b.  Did  the  crew  members  alert  others  to  impending  decisions  and  actions? 

□  Crew  members  aierted  others  to  impending  decisions  and  actions;  supporting  information 
was  actively  solicited  from  other  crew  members 

□  Crew  members  occasionally  solicited  information  from  others  regarding  impending  decisions 
and  actions;  no  significant  compromises  to  flight  safety  or  mission  effectiveness  occurred 

□  Crew  members  did  not  keep  each  other  informed  of  impending  decisions  and  actions; 
compromises  to  flight  safety  or  mission  effectiveness  arose  when  crew  members  waited  for 
others  to  volunteer  significant  information 

Issue  9-2  Solicitation  of  Crew  Assistance 

a.  Did  crew  members  request  assistance  from  others  to  avoid  becoming  overloaded? 

□  Crew  members  consistently  requested  assistance  from  others  before  they  became 
overloaded  or  diverted  their  attention  from  a  critical  task 

□  Crew  members  occasionally  requested  assistance  from  others  when  they  became 
overloaded:  no  significant  compromises  to  flight  safety  or  mission  effectiveness  arose  from 
one  or  more  crew  members  being  unnecessariiy  task  overloaded 
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a  Crew  members  failed  to  request  assistance  from  others  after  becoming  task  overloaded: 
significant  compromises  to  flight  safety  or  mission  effectiveness  occurred  as  a  result 


Basic  Quality  10  Crew  member  actions  mutually  cross-monitored 

Issue  10-1  Scanning  for  Crew  Enx>r 

a.  Did  crew  members  acknowledge  that  human  error  is  a  common  occurrence? 

□  Crew  members  acknowledged  that  crew  error  is  a  common  occurrence,  requiring  active 
involvement  of  all  crew  members  in  detecting  and  breaking  the  error 

□  Crew  members  understood  that  human  errors  can  occur  during  the  mission,  occasionally 
requiring  the  monitoring  and  intervention  by  all  crew  members 

□  Crew  members  believed  or  acted  as  if  human  error  is  a  rare  occurrence 

b.  Did  crew  members  take  responsibility  for  scanning  for  errors  that  might  affect  flight  safety  or 

mission  effectiveness? 

n  When  errors  were  noted,  the  crew  member  committing  the  error  was  quickly  informed  and/or 
assisted  in  a  professional  manner 

o  Crew  members  assumed  some  responsibility  for  monitoring  the  performance  of  others:  no 
significant  compromises  to  flight  safety  or  mission  effectiveness  arose  from  neglect  of  cross 
monitoring 

□  Crew  members  assumed  only  responsibility  for  monitoring  their  own  performance:  significant 
incidents  of  crew  error  remained  undetected  by  others  until  flight  safety  or  mission 
effectiveness  was  significantly  compromised 

c.  Did  crew  members  act  quickly  and  constructively  to  assist  others  in  correcting  errors? 

□  Crew  members  were  constantly  alert  for  crew  errors,  assuming  responsibility  for  monitoring 
their  own  performance  and  that  of  others  for  errors  affecting  flight  safety  or  mission 
effectiveness 

□  When  errors  were  noted,  the  crew  member  committing  the  error  was  informed  and/or 
assisted:  however,  some  improvement  is  possible  in  the  timeliness  or  manner  of  monitoring 

□  When  errors  were  noted,  the  crew  used  the  opportunity  to  insult  or  berate  the  crew  member 
committing  the  error 

Issue  10-2  Two-Challenge  Rule 

a.  Was  the  two-challenge  rule  discussed  prior  to  mission  execution? 

□  The  two-challenge  rule  was  thoroughly  discussed  by  crew  members  prior  to  execution  of 
mission 

□  The  two-challenge  rule  was  acknowledged  by  crew  members  prior  to  mission  execution 

□  Crew  members  ignored  the  two-challenge  rule,  leaving  its  implementation  ambiguously 
defined 

b.  Was  the  two-challenge  rule  implemented  effectively,  if  required? 

o  The  two-challenge  rule  was  effectively  implemented  when  required,  with  minimal 
compromise  to  flight  safety 

□  The  two-challenge  rule  was  implemented  when  required,  but  resulted  in  some  confusion  or 
tension  between  crew  members 
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□  Crew  members  failed  to  implement  the  two-challenge  rule  when  required,  resulting  in 
significant  compromise  of  flight  safety 


Basic  Quality  11  Supporting  infonnation  and  actions  offered  by  crew 

Issue  11-1  Anticipation  and  Offering  of  Required  Informationa. 

a.  Did  crew  members  anticipate  the  need  to  provide  information  and  warnings  to  the  PC  or  pilot  on 

the  controls? 

□  Crew  members  consistently  anticipated  the  need  to  provide  information  or  warnings  to  the 
pilot  on  the  controls  during  critical  phases  of  flight 

□  Crew  members  provided  information  or  warnings  to  the  pilot  on  the  controls  when  requested; 
no  significant  compromises  to  flight  safety  or  mission  effectiveness  arose  due  to  failure  to 
offer  critical  information  or  warnings 

□  Crew  members  failed  to  provide  information  or  warnings  requested  by  PC  or  pilot  on  the 
controls,  or  provided  information  and  warnings  only  reluctantly:  significant  compromises  to 
flight  safety  or  mission  effectiveness  occurred  as  a  result 

b.  Were  information  and  warnings  provided  in  a  timely  manner? 

□  Required  information  and  warnings  were  consistently  provided  In  a  timely  manner 

□  Required  information  and  warnings  were  generally  provided  in  a  timely  manner;  no 
significant  compromises  to  flight  safety  or  mission  effectiveness  arose  due  to  lack  of 
timeliness  of  supporting  Information  or  warnings 

□  Required  information  or  warnings,  when  offered,  were  late;  significant  compromises  to  flight 
safety  or  mission  effectiveness  occurred  as  a  result 

Issue  11*2  Anticipation  and  Offering  of  Required  Assistance 

a.  Did  crew  members  anticipate  the  need  to  provide  task  assistance  to  the  PC  or  pilot  on  the 

controls? 

n  Crew  members  consistently  anticipated  the  need  to  provide  task  assistance  to  PC  or  pilot  on 
the  controls  during  critical  phases  of  flight 

□  Crew  members  provided  task  assistance  to  PC  or  pilot  on  the  controls  when  requested;  no 
significant  compromises  to  flight  safety  or  mission  effectiveness  arose  due  to  failure  to  offer 
assistance 

□  Crew  members  failed  to  provide  task  assistance  requested  by  PC  or  pilot  on  the  controls,  or 
provided  assistance  only  reluctantly;  significant  compromises  to  flight  safety  or  mission 
effectiveness  occurred  as  a  resuit 

b.  Was  the  assistance  provided  in  a  timely  manner? 

□  Required  task  assistance  was  consistently  provided  in  a  timely  manner 

□  Required  task  assistance  was  generally  provided  in  a  timely  manner;  no  significant 
compromises  to  flight  safety  or  mission  effectiveness  arose  due  to  lack  of  timeliness  of 
supporting  assistance 

□  Required  cockpit  task  assistance,  when  offered,  was  late;  significant  compromises  to  flight 
safety  or  mission  effectiveness  occurred  as  a  result 
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Basic  Quality  12  Advocacy  and  assertiveness  practiced 

Issue  12-1  Advocacy 

a.  Did  crew  members  justify  their  recommended  plans  and  courses  of  action  with  appropriate 

rationale? 

□  Time  permitting,  crew  members  consistently  provided  rationale  for  their  recommended  plans 
and  courses  of  action;  a  professional  atmosphere  was  maintained 

□  When  misunderstanding  was  apparent,  crew  members  provided  rationale  for  their 
recommended  plans  and  courses  of  action;  some  level  of  objectivity  was  maintained 

o  Crew  members  frequently  justified  their  recommendations  on  rank  or  experience  levei,  rather 
than  logic;  personaiity  conflicts  resulted  from  this  behavior 

b.  Did  crew  members  request  feedback  to  insure  others  have  correctly  understood  them? 

o  Crew  members  consistently  requested  feedback  to  insure  that  others  had  correctly 
understood  their  statements  or  rationale 

o  Crew  members  requested  feedback  when  it  became  obvious  that  others  had  misunderstood 
their  statements  or  rationale 

□  Crew  members  gave  little  concern  to  insuring  that  others  had  correctly  understood  their 
statements  or  rationale;  misunderstandings  were  permitted  to  continue 

c.  Did  crew  members  each  exhibit  willingness  to  accept  input  from  others? 

□  Time  permitting,  crew  members  practiced  good  listening  habits,  allowing  others  to  state  their 
rationale  before  reacting  to  recommended  plans  or  courses  of  action 

□  Time  permitting,  crew  members  generally  allowed  others  to  explain  their  recommendations 
before  reacting 

□  Crew  members  displayed  a  closed  mind  with  regard  to  accepting  recommendations  from 
others;  decisions  and  actions  were  overly  influenced  by  a  crew  member  who  possessed  a 
dominant  personality 

Issue  12-2  Rank  or  Experience  Intimidation 

a.  Did  the  PC  encourage  junior  crew  members  to  speak  up  during  mission? 

□  PC  actively  promoted  objectivity  by  encouraging  junior  crew  members  to  speak  up 
regardless  of  rank  or  experience  level 

□  PC  tolerated  junior  crew  members  to  speak  up  regardless  of  rank  or  experience  level 

□  PC  used  rank  or  experience  factors  to  impose  authoritarian  controi  over  other  crew  members 

b.  Were  junior  crew  members  reluctant  to  voice  opinions  that  conflicted  with  more  senior  crew 

members? 

o  Junior  crew  members  did  not  hesitate  to  speak  up  when  they  disagreed  with  others;  junior 
crew  members  understood  that  more  experienced  aviators  can  occasionally  commit  errors  or 
lose  situational  awareness 

□  Junior  crew  members  voiced  disagreements  when  asked;  junior  crew  members  spoke  up 
when  they  observed  obvious  errors  being  committed  by  more  experienced  aviators 

□  Crew  members  were  generally  reluctant  to  challenge  a  senior  or  more  experienced  aviator, 
even  when  they  knew  themselves  to  be  correct;  crew  members  generally  assumed  that 
others  knew  what  they  were  doing,  regardless  of  the  facts 
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c.  Did  each  crew  member  display  a  sense  of  responsibility  for  adhering  to  flight  regulations, 

operating  procedures,  and  safety  standards? 

□  Every  crew  member  displayed  a  sense  of  responsibility  for  adhering  to  flight  regulations, 
operating  procedures,  and  safety  standards 

□  Crew  members  spoke  up  when  they  believed  that  flight  regulations,  operating  procedures,  or 
safety  standards  were  being  violated 

□  Crew  members  denied  personal  responsibility  for  flight  safety,  allowing  others  to  violate 
known  flight  regulations,  operating  procedures,  or  safety  standards  because  of  their  rank  and 
experience  level 

Basic  Quality  13  Crevwlevel  after  action  reviews  accomplished 

Issue  13-1  Critique  and  improvement  of  Crew  Performance 

a.  Did  the  crew  critique  major  decisions  and  actions  to  identify  areas  of  improvement? 

□  Crew  critiqued  major  decisions  and  actions,  identifying  options  and  factors  that  should  have 
been  discussed,  and  outlining  ways  of  improving  crew  performance  in  future  missions 

a  Crew  reviewed  major  decisions  and  actions,  focusing  on  obvious  errors,  and  identifying  ways 
of  avoiding  those  errors  in  future  missions 

□  Crew  avoided  any  discussion  of  major  decisions  and  actions;  obvious  errors  were  ignored 
with  little  or  no  concern  about  Improving  crew  performance  in  future  missions 

b.  Was  the  after  action  critique  conducted  in  a  professional  manner,  with  emphasis  on  education 

and  improvement  of  crew  performance? 

□  Critique  of  crew  decisions  and  actions  was  conducted  in  a  professional  manner;  finger¬ 
pointing  was  avoided,  with  emphasis  on  education  and  improvement  of  crew  performance 

□  Critique  of  crew  decisions  and  actions  avoided  personality  conflicts  or  other  attitudes  that 
would  detract  from  the  discovery  of  improved  procedures 

□  After  action  reviews  consisted  of  finger  pointing;  little  or  no  collaborative  spirit  was  exhibited; 
the  crew  appears  likely  to  repeat  poor  performance 
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BATTLE-ROSTERED  CREW  EVALUATION/TRAINING  GRADE  SUP 
For  use  of  this  form,  see  Aircraft  ATM;  the  proponent  agency  is  TRADOC 

BATTLE-  NAME  RANK 

ROSTERED  PC: 

CREW  — - 

EXAMINEES/  _ 

TRAINEES  NONRATED  CREW  MEMBERS 

DUTY  SYMBOL  NAME  RANK 


EVALUATOR/  NAME  RANK 

INSTRUCTOR 


UNIT: 


CREW  DATA 


TOTAL  BATTLE-ROSTERED 

CREW  HOURS 

DATE  DESIGNATED  A  BATTLE- 
ROSTERED  CREW: 

PURPOSE:  EVALUATION/TRAININC 

l  TIME  TODAY: 

CUMULATIVE  TIME: 

TYPE  AIRCRAFT: _ 

CREW  TASK  1 _ D/N/NVD  CREW  TASK  6  _ D/N/NVD 

CREW  TASK  2 _ D/N/NVD  CREW  TASK  7  _ D/N/NVD 

CREW  TASK  3 _ D/N/NVD  CREW  TASK  8  _ D/N/NVD 

CREW  TASK  4 _ D/N/NVD  CREW  TASK  9  _ D/N/NVD 

CREW  TASK  5 _ D/N/NVD  CREW  TASK  1 0 _ D/N/NVD 

DAY  NIGHT  WX  SIMULATOR  NVG  NVS 


_ EVALUATOR/INSTRUCTOR  RECOMMENDATIONS _ 

□  (ISSUE)  (VALIDATE)  CREW  QUALIFICATIONS 

□  (SUSPEND)  (REVOKE)  CREW  QUALIFICATIONS 

□  REQUIRES  ADDITIONAL  (FLIGHT)  (ACADEMIC)  (SIMULATION  DEVICE)  TRAINING 
a  SEE  BACK  FOR  COMMENTS 

I  HAVE  DEBRIEFED  THE  EXAMINEES/TRAINEES  AND  INFORMED  THEM  OF  THEIR  STATUS. 

EVALUATOR'S/INSTRUCTOR'S  SIGNATURE: _ 

WE  HAVE  BEEN  DEBRIEFED  BY  THE  EVALUATOR/INSTRUCTOR  AND  UNDERSTAND  OUR  CURRENT 
STATUS. 

PC’S  SIGNATURE: _ 

Prs  SIGNATURE: _ 

NONRATED  CREW  MEMBER’S  SIGNATURES:  _ _ _ 


OVERALL  GRADE  FOR  THIS  FLIGHT  IS:  S  U  NA  DATE: 


DA  FORM  7121-R,  MAR  92 
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Appendix  D 

Mission  Performance  Measures 


Terrain  Flight  Navigation —  Measured  during  segments  3  &  4  of  the 
mission,  depending  on  .the  scenario  being  flown. 


Nhat  to  Measure 

Number  (N)  of 
deviations  from  the 
corridor  due  to 
misorientation 

How  to  Collect 

FS  printout,  page 

25,  "Cross  country 
map"  (12  X  12  K  or 

24  X  24  K)  with 
ground  track  trace; 
verified  by  video 
tape  review 

Measurement 

Parameters 

Sum  (N) 

Distance  of 
deviation  outside  of 
corridor  due  to 
misorientation 

FS  printout,  page 

25,  "Cross  country 
map"  (12  X  12  K  or 

24  X  24  K)  with 
ground  track  trace; 
verified  by  video 
tape  review 

Sum  (N)  <500m 

Sum  (N)  >500m  <1500m 
Sum  (N)  >1500m 

Deviation  (seconds) 
from  required  time 
of  arrival  at 
landing  zone 

Live 

observation/time  on 
tape 

Actual  time  compared 
to  time  designated 
in  OPORD/FRAG 

Number  (N)  of 
mission  (route) 
segments  completed 

Live  observation; 

FS  printout 

Sum  (N) 

Time  (sec)  to  fly 
each  mission  segment 

Live 

observation/time  on 
tape 

Elapsed  time  from  PZ 
to  LZ  and  return  to 

PZ 
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Threat  avoidance/evasion  —  Scenario  will  include  encounters  with 
different  enemy  anti-aircraft  systems.  Different  aural  warnings 
in  the  cockpit  warn  of  the  mode  of  the  threat  weapon  system: 
search,  track,  or  missile.  The  three  modes  represent  increasing 
levels  of  threat  to  the  crew. 


What  to  Measure 

How  to  Collect 

Measurement 

Parameters 

Threat  avoidance- 
number  of 

activations  (N)  of 
track  or  missile 
aural  warnings 

Live  observation 

Sum  (N) 

Threat  evasion- 
duration  (sec)  of 
track  or  missile 
aural  warnings 

Live  observation 

Elapsed  time  from 
beginning  to 
cessation  of  track 
or  missile  aural 
warning 

Number  of  track  or 
missile  aural 
warnings  (N^) 
exceeding  10  sec 

Live  observation 

Sum  (N^) 

Outcome  of  threat 
encounter 

Live  observation; 
video  tape  review 

Broke  lock,  took 
hits,  crashed,  or 
misoriented 

Aircraft  emergencies  —  Two  aircraft  malfunctions  will  occur 
during  the  mission.  One  major  malfunction  will  require  immediate 
crew  action,  whereas  the  other  malfunction  will  be  less  severe 


and  result  in  a  slow, 
system. 

gradual  change  in  the 

status  of  an  aircraft 

What  to  Measure 

How  to  Collect 

Measurement 

Parameters 

Diagnosis  of  major 
emergency 

Live  observation; 
Evaluator  worksheet 

Correct  diagnosis  of 
emergency?  (Yes/no) 

Proper  adherence  to 
emergency  procedures 

Live  observation; 
Evaluator  worksheet 

Number  of  steps 
accomplished  versus 
number  required 

Eventual  outcome 

Live  observation; 
Evaluator  worksheet 

Normal,  hard,  or 
crash  landing 

Time  to  detect  minor 
malfunction 

Live 

observation/time  on 
tape;  Evaluator 
worksheet 

Elapsed  time  (sec) 
to  discover 
malfunction 
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Unexpected  event  —  The  unexpected  event  (inadvertent  IMC)  will 
lead  to  the  instriament  recovery,  an  NDB  approach. 


What  to  Measure 

How  to  Collect 

Measurement 

Parameters 

Proper  adherence  to 
recovery  procedures 

Live  observation; 
Evaluator  worksheet 

Number  of  VHIRP 
steps  accomplished 
versus  the  number 
required 

Eventual  outcome 

Live  observation 

Successful 
transition  from  VMC 
to  IMC?  (Yes/no) 
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Instrument  recovery  —  Crews  will  perform  an  NDB  approach  for 
recovery.  The  approach  will  be  initiated  following  entry  into 
inadvertent  IMC.  Measurements  will  be  taken  for  both  the 
planning  and  execution  portions  of  the  approach. 


Approach  planning  measurements  — 


What  to  Measure 


How  to  Collect 


Measurement 

Parameters 


Crew  planning, 
discussion,  and 
rehearsal  of 
essential  elements 
of  the  approach 
prior  to  executing 
the  approach 


Live  observation 


Both  crewmembers 
review,  discuss,  and 
rehearse  the 
approach  (Value  =  3) 

One  crewmember 
reviews  the 
approach,  briefs  the 
other  prior  to 
executing  the 
approach,  then  talks 
him  through  it 
(Value  =  2) 

One  crewmember 
reviews  the  approach 
and  talks  the  other 
through  it 
(Value  =  1) 

One  crewmember 
reviews  and  executes 
the  approach  with  no 
assistance  from  the 
other  (Value  =  0) 
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Approach  execution  measurements  — 


What  to  Measure 

Courses  and  headings 
flown  during  the 
recovery 

How  to  Collect 

FS  printout,  page 

32,  "Approach  map" 

(12  X  12  K)  with 
ground  track  trace; 
Evaluator  worksheet 

Measurement 

Parameters 

Number  of  deviations 
exceeding  +/-  5 
degrees 

Altitudes  during  the 
recovery 

Live  observation; 
Evaluator  worksheet 

Number  of  deviations 
exceeding  +/-  100 
feet 

Timing  of  the 
inbound  leg 

Live  observation; 
Evaluator  worksheet 

Did  crew  properly 
time  the  inbound  leg 
of  the  approach? 
(Yes/no) 

Eventual  outcome 

Live  observation 

Following  transition 
from  IMC  to  VMC,  was 
aircraft  in  position 
to  safely  land? 
(Yes/no) 

Descent  to  minimum 
altitude 

Live  observation;  FS 
printout;  video  tape 
review 

Did  crew  descend 
below  MDA  at  correct 
time?  (Yes/No) 
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Mission  threatening  crew  error  —  There  may  be  cases  when  crew 
error  leads  to  an  unanticipated  aircraft  crash,  probable  aircraft 
damage,  or  other  disruptive  events.  These  instances  will  be 
evaluated  on  a  case  by  case  basis  to  determine  their  effect  on 
mission  performance. 


What  to  Measure 


Degree  of  adverse 
effect  on  the 
mission  caused  by 
unanticipated  crew 
error 


Number  fNl  of  crews 
completing 
mission (s) 


How  to  Collect 


Video  tape  review 


Measurement 

Parameters 

No  occurrence  of 
mission  threatening 
crew  error  (Value  = 
4) 

Crew  error  results 
in  no  aircraft 
damage  and  mission 
fully  complete  with 
precise  timing  and 
accuracy  (Value  =  3) 

Crew  error  results 
in  minor  aircraft 
damage  and/or  minor 
deviation  of  mission 
completion  within 
time  and  accuracy 
(Value  =  2) 

Crew  error  results 
in  temporary  loss  of 
aircraft  and/or  crew 
and  mission 
marginally  complete 
due  to  recoverable 
time  or  accuracy 
failures  (Value  =  1) 

Crew  error  would 
likely  result  in 
minor  damage  to 
aircraft  and/or  crew 
and/or  mission  is 
incomplete  due  to 
grass  time  or 
accuracy  failures 
(Value  =  0) 


Live  observation; FS 
printout;  video  tape 
review 


Sum  (N) 
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Appendix  E 
Exit  Interviews 


Appendix  E-1 


Aircrew  Coordination  Training  Validation  Testbed 
Evaluator  and  Trainer  Exit  Interview 


I .  Introduction 

This  form  is  to  be  used  as  general  guidance  in  structuring 
the  debrief  for  the  Evaluator  and  Trainer  participants  in  the 
aircrew  coordination  validation  testbed,  August  1992.  Questions 
are  meant  to  be  suggestive  and  should  not  restrict  your  answers. 
Some  of  the  questions  may  overlap  with  others  or  seem  repetitive, 
but  we  need  to  make  sure  that  all  the  issues  are  covered.  The 
order  of  the  questions  and  answers  is  unimportant.  You  may  skip 
a  question  if  you  answered  it  on  a  previous  question.  The  entire 
set  of  questions  will  be  discussed  with  the  Evaluators  and 
Trainers.  The  debrief  forum  will  involve  both  individual  and 
group  interviews  according  to  the  published  schedule. 


Important 

•  Participants  must  bring  the  Aircrew  Coordination 
Exportable  Training  Package  materials  to  the 
debriefing. 

•  It  is  suggested  that  the  questions  be  provided  to  the 
interviewees  prior  to  the  scheduled  interview. 

•  There  are  no  "right”  or  "wrong"  answers.  We  are  asking 
for  your  honest  opinions  so  that  we  can  improve  the 
testbed  methods  and  materials. 
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II.  Course  of  Instruction 

1.  Has  adequate  time  (or  too  much  time)  been  allocated  for  each 
segment  of  the  course?  In  answering  this  question,  consider' 
both  the  trainer  course  and  the  student  course. 

2 .  Was  the  number  of  video  segment  case  studies  adequate  to 
teach  the  13  basic  qualities? 

3.  Were  the  written  case  studies  effective  in  emphasizing 
teaching  points?  If  yes,  was  the  number  of  written  case 
studies  adequate  to  teach  the  13  basic  qualities? 

4.  Has  the  course  tied  crew  coordination  principles  and 
techniques  adequately  to  the  ATMs? 

5.  How  many  simulator  sessions  are  required  to  achieve  crew 
coordination  proficiency? 

6.  What  affect,  if  any,  did  the  pre-training  evaluation  mission 
in  the  simulator  have  on  the  classroom  instruction  part  of 
the  course?  Should  each  crew  be  required  to  complete  a 
"crew  coordination”  simulator  session  prior  to  the  classroom 
instruction? 

7.  Would  read-ahead  packages  enhance  the  flow  of  the  course?  If 
so,  what  material  would  you  suggest  be  included? 

8.  Overall,  did  the  trainer  course  adequately  prepare  you  to 
teach  the  aviator  student  course? 

9.  Are  there  any  trainer  course  segments  (for  example,  MOI, 
evaluation,  scenario  familiarization)  that  should  receive 
more  or  less  emphasis? 

III.  Scenarios  Used  During  the  Evaluation  Phase 

1.  Were  the  evaluation  scenarios  of  about  the  correct  level  of 
difficulty? 

2 .  Was  the  scenario  reasonably  realistic  in  terms  of  the  types 
of  missions  these  aviators  must  execute? 

3 .  Was  there  enough  pre-mission  planning  time  for  the  crews? 

4.  Do  you  have  any  suggestions  for  improving  the  preflight  air 
mission  briefing  and  crew  mission  briefings? 

5.  Do  you  have  any  suggestions  for  improving  the  post-flight 
crew-level  after-action  review? 

6.  Do  you  have  any  suggestions  for  improving  the  post-flight 
evaluator  debriefing? 
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7. 


What  general  comments  did  the  aircrews  make  that  might  help 
us  improve  the  scenario? 

8.  Do  you  have  any  specific  suggestions  for  improving  the 
scenario  and/or  the  mission  objectives? 

9.  Do  you  have  any  specific  suggestions  for  improving  the 
scenario  by  adding  or  deleting  tasks? 

10.  Did  the  scenarios  allow  adequate  demonstration  and 
observation  of  the  13  basic  qualities  associated  with  crew 
coordination? 

11.  If  the  answer  to  item  10  above  is  no,  what  tasks  or  mission 
events  should  be  added  to  permit  adequate  demonstration  and 
observation  of  each  basic  quality? 


IV.  Evaluator's  Workbook 

1.  How  did  you  utilize  the  Evaluator's  Workbook; 

a.  During  pre-mission  activities? 

b.  During  the  flight? 

c.  During  post-mission  activities? 

d.  After  the  mission  was  completed  (grading  and  rating)? 

2 .  Regarding  the  videotapes : 

a.  Did  you  review  then  evaluate,  or  did  you  review  and 
evaluate  at  the  same  time? 

b.  Did  you  use  the  videotapes  to  review  specific  areas  of 
the  tape  where  you  thought  you  missed  important 
information? 

c.  Did  you  review  the  whole  tape? 

d.  What  general  comments  did  the  aircrews  make  as  they 
observed  their  videotape? 

3 .  What  elements  of  the  Evaluator  Worksheets  were  helpful  or 
not  helpful,  and  why? 

a.  Segment  description? 

b .  ATM  Task? 

c.  ATM  Task  performance? 

d.  Segment  overall  performance? 
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e.  Aircrew  Coordination  Basic  Qualities? 

f.  Evaluator's  Notes? 

4.  Were  there  elements  in  the  Workbook  that  were  confusing?  If 
so,  which? 

5.  Are  there  some  elements  in  the  Workbook  that  you  could  do 
without?  If  so,  which? 

6.  Are  there  any  additional  elements  that  you  would  like  to 
add? 

7.  Do  you  have  any  other  suggestions  for  improving  the  Workbook 
(be  specific)? 


V.  Aircrew  Coordination  Basic  Qualities 

1.  Can  any  of  the  13  basic  qualities  be  combined  or  eliminated? 

a.  If  two  or  more  qualities  are  proposed  for  combining, 
give  specific  examples  of  how  they  overlap.  [Remember: 
Basic  qualities  do  not  have  to  be  completely  different 
from  one  another.] 

b.  If  a  basic  quality  is  proposed  for  elimination,  give 
specific  reasons  why  this  should  not  be  evaluated  as 
part  of  crew  coordination  performance.  Also  state  where 
this  aspect  of  performance  would  be  evaluated,  if  not 
under  crew  coordination. 

2.  How  were  the  behavioral  rating  anchors  (text  descriptions) 
useful  or  not  useful  to  you  in  achieving  reliable  and 
objective  ratings  of  crew  performance?  Please  provide 
specific  examples,  if  possible. 

3 .  Were  some  of  the  behavioral  rating  anchors  more  useful  than 
others?  If  so,  which  ones  were  found  to  be  more  useful? 
Which  ones  were  found  to  be  less  useful?  Please  provide 
specific  examples  of  how  you  attempted  to  use  the  behavioral 
rating  anchors. 

4.  How  were  the  behavioral  evaluation  factors  (bulletized 
descriptions)  useful  or  not  useful  to  you  in  achieving 
reliable  and  objective  ratings  of  crew  performance?  Please 
provide  specific  examples,  if  possible. 

5.  Were  some  of  the  behavioral  rating  factors  more  useful  than 
others?  If  so,  which  ones  were  found  to  be  more  useful? 
Which  ones  were  found  to  be  less  useful?  Please  provide 
specific  examples  of  how  you  attempted  to  use  the  behavioral 
rating  factors. 
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6.  How  did  you  use  the  evaluation  factors?  Did  you  use  them  to 
instruct  in  the  classroom  or  simulator?  Crew  debriefing? 

7.  Should  we  continue  to  have  both  the  factors  and  anchors  for 
the  evaluation  guidance?  If  we  use  only  one,  which  one 
would  you  choose? 

8.  Do  the  audio  segments  provide  adequate  opportunity  for 
practicing  your  application  of  the  rating  scales? 

a.  Were  the  13  basic  qualities  addressed  in  an  adequate 
fashion?  If  not,  how  could  the  segments  be  expanded  to 
address  each  basic  quality? 

b.  Would  you  find  video  tapes  to  be  more  useful?  If  so, 
what  type  of  vignettes  would  you  recommend  be  included? 

9.  Was  the  7-point  scale  a  good  choice  for  rating  crew 
coordination  basic  qualities?  Were  the  descriptors  (very 
poor,  poor,  marginal,  adequate,  acceptable,  good,  very  good, 
superior)  for  each  number  helpful?  Do  you  have  any 
suggestions  for  improvement? 

10.  Think  back  to  each  of  the  13  basic  qualities.  Each  has  a 
behavioral  anchor  for  numbers  1,  4,  and  7. 

a.  Were  the  behavioral  anchors  helpful? 

b.  What  are  your  suggestions  for  improving  them  (be  as 
specific  as  possible  so  we  can  incorporate  your 
suggestions) ? 

11.  Think  back  to  each  of  the  13  basic  qualities.  Each  has 
evaluation  factors  for  numbers  1,  4,  and  7. 

a.  Were  the  evaluation  factors  helpful? 

b.  What  are  your  suggestions  for  improving  them  (be  as 
specific  as  possible)? 

12.  Were  you  reluctant  to  give  crews  ratings  below  "fully 
acceptable"?  If  yes,  why? 

13.  If  you  had  experience  using  both  the  behavioral  anchors  and 
the  evaluation  factors,  do  you  think  that  one  or  the  other 
methods  influenced  you  to  give  higher  or  lower  grades?  For 
example,  if  you  used  the  evaluation  factors,  do  you  think 
that  led  you  to  give  higher  or  lower  grades  than  if  you  had 
used  the  behavioral  anchors? 

14.  How  often  did  you  refer  to  the  explanations  in  the 
behavioral  anchors? 

15.  How  often  did  you  refer  to  the  evaluation  factors? 
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16.  Did  the  frequency  of  referral  to  the  behavioral  anchor 
descriptions  change  over  time  (e.g.,  less  referral  to  the 
anchor  descriptions  with  experience) ? 

17.  Did  the  frequency  of  referral  to  the  evaluation  factors 
change  over  time  (e.g.,  less  referral  to  the  factor 
descriptions  with  experience)? 

18.  Are  there  any  aspects  of  crew  performance  not  adequately 
covered  in  the  13  basic  qualities? 

a.  If  so,  would  you  address  these  aspects  within  the 
context  of  one  of  the  existing  basic  qualities?  Please 
be  specific. 

b.  If  so,  would  you  address  these  aspects  as  a  separate 
basic  quality?  Please  be  specific. 


VI.  Modified  Grade  Slips 

1.  Was  the  format  of  the  grade  slip  understandable  and  easily 
used?  Any  specific  suggestions  for  improvement? 

2.  Did  you  weight  flying  skills  and  aircrew  coordination  skills 
differently?  Did  you  give  them  equal  weight? 

3.  Was  the  satisfactory  plus  (S+) ,  satisfactory  (S) ,  and 
satisfactory  minus  (S-)  grading  system  helpful?  Would  you 
like  to  use  S+,  S,  and  S-  for  APART  rides? 

4.  Would  you  have  liked  to  use  the  basic  quality  notations 
(1,2,  .  .  .13)  for  both  positive  and  negative  crew 
coordination  behaviors?  If  yes,  would  using  a  "+"  or 

sign  next  to  each  BQ  associated  with  ATM  Task  performance  be 
a  reasonable  marking  technique,  or  do  you  think  it  would  be 
too  complex? 

5.  When  you  gave  an  overall  mission  grade,  what  were  your 
criteria? 

6.  Did  you  find  the  Comment  Slip  useful?  If  not,  do  you  have 
any  specific  suggestions  to  improve  its  use? 


VII.  General  Observations 

1.  What  is  your  overall  impression  of  the  adequacy  of  the 
aircrew  coordination  training  provided?  Do  you  have  any 
recommendations  for  improvement? 

2.  What  is  your  overall  impression  of  the  adequacy  of  the 
evaluation  training  provided?  Do  you  have  any 
recommendations  for  improvement? 
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3.  What  is  your  overall  impression  of  the  conduct  of  the 
aircrew  coordination  evaluations?  Do  you  have  any 
recommendations  for  improvement? 

4.  If  this  training  package  is  fielded  Army-wide,  should  there 
be  a  system  to  track  testbed  participants  in  longitudinal 
studies  of  the  Army's  crew  coordination  program? 

5.  What  effect  has  participation  in  this  project  had  on  you 
personally? 

6.  Do  you  have  any  questions  or  concerns  that  you  would  like  to 
ask  or  convey  to  the  crew  coordination  staff? 


E-1-7 


E-1-8 


Appendix  E-2 


Aircrew  Coordination  Training  Validation  Testbed 
Crewmember  Exit  Interview 


I .  Introduction 

This  form  is  to  be  used  as  general  guidance  in  structuring 
the  debrief  for  the  crewmember  participants  in  the  aircrew 
coordination  validation  testbed,  August  1992.  Questions  are 
basically  conversation  starters  and  are  not  meant  to  limit  free 
discussion;  follow  where  the  comments  may  lead  and  develop  points 
as  necessary.  The  debriefing  forum  will  involve  small  group 
interviews  according  to  the  published  schedule. 


Important 

Participants  must  bring  the  Aircrew  Coordination 
Student  Course  Outline  and  Reference  Book  (Volume  2)  to 
the  debriefing. 

It  is  suggested  that  the  questions  be  provided  to  the 
interviewees  prior  to  the  scheduled  interview. 

There  are  no  "right"  or  "wrong"  answers.  We  are  asking 
for  your  honest  opinions  so  that  we  can  improve  the 
testbed  methods  and  materials. 
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II .  Course  Administration 

1.  Was  the  classroom  appropriately  arranged  for  the  number  of 
students  present? 

2.  Was  the  number  of  students  in  the  class  about  the  right  size 
for  this  training? 

3.  Did  the  videotaping  of  the  classroom  instruction  detract 
from  or  enhance  the  classroom  environment?  If  so,  how? 

4.  Was  the  instructional  staff  properly  prepared  to  conduct  the 
course?  If  not  prepared,  what  deficiencies  did  you  note? 

5.  Were  the  facilities  adequate  during  the  simulator  phase  of 
training?  If  not,  what  was  unsatisfactory? 

6.  What  changes  do  you  recommend  to  improve  the  administration 
of  future  courses? 

III .  Course  Structure 

1.  Was  the  course  well  organized  in  terms  of  subject  flow: 

a.  overview? 

b.  history? 

c.  structure  of  the  course? 

d.  crew  coordination  model? 

e.  crew  coordination  elements? 

f.  basic  qualities? 

g.  crew  coordination  objectives? 

h.  definition,  discussion,  effectiveness  factors,  and 
examples  of  basic  qualities? 

i.  phase  review? 

j .  hands-on  simulator  application? 

2.  Were  the  subjects  presented  applicable  to  your  job  as  an 
Army  aviator? 

3.  Were  the  subjects  well  developed  so  that  you  are  confident 
that  you  understand  the  material? 
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4. 


Was  the  interrelationship  between  crew  coordination  elements 
in  the  ATM  Tasks,  the  basic  qualities,  and  the  crew 
coordination  objectives  clearly  established? 

5.  Were  the  basic  qualities  well  defined  and  explained? 

6.  Were  the  effectiveness  factors  clearly  linked  to  the  basic 
qualities? 

7.  Were  the  crew  coordination  objectives  well  demonstrated  in 
terms  of  case  studies? 

8.  Were  the  crew  coordination  objectives  well  demonstrated  in 
terms  of  video  segments? 

9.  Was  there  about  the  right  mix  of  video  and  written  case 
studies  to  help  you  understand  the  basic  qualities  and  crew 
coordination  objectives? 

10.  Was  the  Student  Handout  satisfactory? 

a.  Did  it  assist  you  in  following  the  instructor's 
presentation? 

b.  Was  there  sufficient  white  space  for  taking  notes? 

c.  Should  any  other  items  be  added? 

11.  Did  you  read  any  of  the  articles  in  the  Reference  Book?  If 
yes,  which  ones  and  were  they  informative? 

12.  Was  the  mission  planning  and  rehearsal  practical  exercise 
helpful?  How  could  it  be  improved? 

13.  Was  the  communications  practical  exercise  helpful?  How 
could  it  be  improved? 

14.  Did  you  complete  the  Hazardous  Thought  Pattern  exercise?  If 
not,  do  you  plan  to  do  so?  If  you  did,  was  it  helpful? 

15.  Did  you  complete  the  Stress  Management  exercise?  If  not,  do 
you  plan  to  do  so?  If  you  did,  was  it  helpful? 

16.  Was  the  course  the  right  length  to  teach  crew  coordination? 
If  not,  what  adjustments  are  necessary? 


IV.  Flight  Simulator  (hands  on) 

1.  Was  the  purpose  of  the  simulator  phase  explained? 

2.  Was  the  hands-on  phase  necessary  to  effectively  teach  crew 
coordination  principles? 
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3.  Was  the  "crawl -walk-run"  approach  to  the  training  and 
evaluation  rides  effective?  More  rides  needed?  Adequate 
number  of  rides?  Too  many  rides? 

4.  Did  you  have  enough  time  during  the  hands-on  periods;  that 
is,  pre-mission  planning  and  rehearsal,  mission  execution, 
and  after-action  review? 

5.  You  participated  in  four  simulator  rides  during  the  testbed. 
What  did  you  think  about  these  rides  in  terms  of  whether 
they  reflect  the  types  of  missions  you  fly?  In  terms  of  the 
level  of  difficulty? 

6.  Do  you  feel  that  the  use  of  the  videotape  of  your  crew's 
simulator  flight  during  the  instructor  debriefing  was  a  good 
training  technique?  Why? 


V.  General  Observations 

1.  What  is  your  overall  impression  of  the  adequacy  of  the 
aircrew  coordination  training  provided?  Do  you  have  any 
recommendations  for  improvement? 

2.  What  is  your  overall  impression  of  the  conduct  of  the 
aircrew  coordination  evaluations?  Do  you  have  any 
recommendations  for  improvement? 

3.  Do  you  personally  feel  that  you  are  now  better  prepared  to 
perform  as  an  Army  aviation  crewmember?  Why? 

4.  Would  you  recommend  that  this  course  be  attended  by  every 
Army  aviator?  Why? 

5.  If  this  training  package  is  fielded  Army-wide,  should  there 
be  a  system  to  track  testbed  participants  in  longitudinal 
studies  of  the  Army's  crew  coordination  program? 

6.  What  effect  has  participation  in  this  project  had  on  you 
personally? 

7.  Do  you  have  and  questions  or  concerns  that  you  would  like  to 
ask  or  convey  to  the  crew  coordination  project  staff? 
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